I have not blogged much recently. My database work has been distracted by Java programming and a crash course in DevOps, leaving me with little worth posting that a tweet couldn’t exhaustively cover.

That changes now. Time for a new blog series that I will call The Data Driven Drinker. I will acquire, imbibe and comment on alcoholic beverages and I hope you will join me. Expect a focus on scotch whisky, on local (Vermont, USA) products and on everything else interesting I taste.

While reading The Security Dialogue, I noticed the code contest and decided to give it a shot.  Here I present a way, one way of many, to solve it.  I enjoy solving newspaper cryptograms but I don’t claim to have any real cryptanalytic experience of any kind so take everything with a big grain of salt.

Given the following ciphertext, and assuming a simple substitution cipher:

jdc9)c9)4ds)9sz21x)z2xs)z214s94!))ud25vx)-25)es4)4dc9)8ced4)q1x)zq1)stqcv)ts)q4)9z8c6s1vfc1e[etqcv!z2t)wc894@)-25)7cvv)es4)q)}+&)ecw4)zq8x)42)Gtq=21!z2t!))jdq1f9)w28)3vq-c1e!

How to crack it?

First make some assumptions. At some point if you don’t get the unencrypted cleartext you may need to revisit these assumptions, but you have to start somewhere.  Knowing your target makes breaking codes much, much easier.  Sometimes you will gain more by spending a few hours researching rather than staring at the cipher.

I made the following assumptions:

• Scriven truthfully relayed that he used a substitution cipher
• The message consists of one or more grammatically proper sentences in English.

Start by counting the frequency of each symbol in the ciphertext. You can do that manually with a message this short but I wrote some basic Perl code to do it.  Run the code, paste in the ciphertext and hit ctrl-D to end, and it prints the character frequencies:

#!/usr/bin/perl 

until(eof(STDIN)) { $ch = getc(STDIN) }
  continue { $ch !~ m/\n/ && ($c{$ch} = defined($c{$ch}) ? $c{$ch}+1 : 1) }

print "$_\t$c{$_}\n" foreach (reverse sort {$c{$a} <=> $c{$b}} (keys %c));
)       31
c       12
4       11
2       11
q       10
s       9
1       9
z       8
9       8
v       7
e       7
t       6
d       6
8       5
!       5
x       5
5       3
-       3
w       3
f       2
j       2
}       1
G       1
&       1
+       1
3       1
6       1
[       1
@       1
u       1
7       1
=       1

The ‘)’ character appears 19 more times than any other symbol in the message and seems distributed throughout the message in a way that it could represent a blank space between words. I will assume for now that ‘)’ = ‘ ‘. Having ciphertext broken up into words makes the rest of the work infinitely easier, so rewrite the message with this change.

jdc9 c9 4ds 9sz21x z2xs z214s94!  ud25vx -25 es4 4dc9 8ced4 q1x zq1 stqcv ts q4 9z8c6s1vfc1e[etqcv!z2t wc894@ -25 7cvv es4 q }+& ecw4 zq8x 42 Gtq=21!z2t!   jdq1f9 w28 3vq-c1e!

Of interest when you do this, ‘))’ appears twice, both times preceded by ‘!’. Going from the assumption that ‘)’ = ‘ ‘, this could indicate what we in the US currently call “French spacing”, or using two spaces after the end of a sentence instead of just one. Though considered deprecated in American English style guides, many people still use it (including me), and autocorrect on mobile devices even takes advantage of that to turn a double tap on the space bar into a period followed by a space and then a capital letter. This adds strength to the assumption and indicates we likely have three sentences. I don’t yet have a reason for why the assumed sentence-terminator ‘!’ sometimes appears in a word, but I will go with it for now.

With the ciphertext letter frequencies in hand, now you need English text letter and word frequencies. You can use ETAOIN SHRDLU as a mnemonic for the most frequently used letters in descending order if you want to keep things simple.

Look at the (assumed) words in the ciphertext. Make lists of all the words with only one letter, only two letters, only three letters, only four letters.  Note any that appear twice or more, and any repeated strings. I made this list by hand but you can write code to do it.

1 letter words: q
2 letter words: c9 ts q4 42
3 letter words: 4ds -25 es4 q1x zq1 -25 es4 w28
4 letter words: jdc9 z2xs 4dc9 7cvv ecw4 zq8x

No repeated 2 letter words
Repeated 3 letter words: -25 es4
No repeated 4 letter words

Repeated digrams (2 letters): c9 jd 4d z2 25 1x zq q1 21     
Repeated trigrams (3 letters): -25 es4 dc9 c1e z2t
Repeated fourgrams (4 letters): !z2t

Repeated letters: vv

Notice the single one letter word: q. In English this can only mean one of the words “I” or “a”. The frequency of ‘q’ in the ciphertext also indicates a possible vowel.

Look for repeated digraphs, pairs or triplets of symbols that appear next to each other frequently. I already noticed ‘!))’ which may mean ‘.  ‘, but I also see ‘c9′ three times. Twice it ends a four letter word, once it stands alone as a two letter word. The ciphertext starts with “jdc9 c9 4ds”, or a four letter word followed by a two letter word made up from the last two letters of the preceding word. In English, “This is” or “What at” or “That at” or even “Shit it” all fit that pattern and can fit grammatically at the start of a sentence. The ‘d’ in the third (three letter) word yields the cleartext ‘h’ in each case, as the second letter of “this”, “what” and “shit”.  Many three letter words have ‘h’ as their second letter and can fit in the sentence I have so far: “This is why”, “This is the”, “What at the”.  I will throw out “That at” for now because I don’t like to see both ‘j’ and ‘9’ meaning ‘t’, unless he decided to sneakily use different symbols for the upper and lowercase versions of the same letter.

So assume for now with some confidence:

')' = ' '
'!' = '.'
'd' = 'h'

For clarity, when I rewrite the text with my substitutions, I will use capital letters for cleartext and lowercase letters for ciphertext (though the ciphertext contains a single capital ‘G’ that I will ignore for the moment).  Rewrite the text with the three substitutions so far:

jHc9 c9 4Hs 9sz21x z2xs z214s94.  uH25vx -25 es4 4Hc9 8ceH4 q1x zq1 
stqcv ts q4 9z8c6s1vfc1e[etqcv.z2t wc894@ -25 7cvv es4 q }+& ecw4 
zq8x 42 Gtq=21.z2t.   jHq1f9 w28 3vq-c1e.

I wrote some simple Perl code to handle rewriting the ciphertext pasted into it, configurable by adding new substitutions to the code.  I will use this going forward instead of substituting manually.

#!/usr/bin/perl

$subst{')'} = ' ';
$subst{'!'} = '.';
$subst{'d'} = 'H';
# add more substitutions here following the same pattern

until(eof(STDIN)) { $ch = getc(STDIN) }
  continue { print defined($subst{$ch}) ? $subst{$ch} : $ch }

Time now to make some guesses.  Earlier I suspected the first two words may encode “What at” or “This is”, and I also know that ‘q’ must represent ‘a’ or ‘I’, so let’s have a look at the ciphertext with those changes. As a simple substitution cipher, no cleartext character can come from two different ciphertext characters, so assume ‘q’ means ‘I’ if ‘c’ means ‘a’, and vice versa (since both ‘c’ and ‘q’ cannot map the same letter).

All use the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'

"What at":

'j' = 'W'
'c' = 'A'
'9' = 'T'
'q' = 'I'

WHAT AT 4Hs Tsz21x z2xs z214sT4.  uH25vx -25 es4 4HAT 8AeH4 I1x zI1 
stIAv ts I4 Tz8A6s1vfA1e[etIAv.z2t wA8T4@ -25 7Avv es4 I }+& eAw4 
zI8x 42 GtI=21.z2t.  WHI1fT w28 3vI-A1e.

"Shit it":

'j' = 'S'
'c' = 'I'
'9' = 'T'
'q' = 'A'

SHIT IT 4Hs Tsz21x z2xs z214sT4.  uH25vx -25 es4 4HIT 8IeH4 A1x zA1 
stAIv ts A4 Tz8I6s1vfI1e[etAIv.z2t wI8T4@ -25 7Ivv es4 A }+& eIw4 
zA8x 42 GtA=21.z2t.  SHA1fT w28 3vA-I1e.

"This is":

'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'

THIS IS 4Hs Ssz21x z2xs z214sS4.  uH25vx -25 es4 4HIS 8IeH4 A1x zA1 
stAIv ts A4 Sz8I6s1vfI1e[etAIv.z2t wI8S4@ -25 7Ivv es4 A }+& eIw4 
zA8x 42 GtA=21.z2t.  THA1fS w28 3vA-I1e.

Each of these seems like a start on a solution.  Where to go from here to give some weight to one choice or the other?  All three could produce a grammatical sentence given the first two words, though I’ve lost faith in “Shit it” at this point if I ever had any.

Take a look at the words where you almost have all of the letters translated, but not quite.  I see the original word “4dc9″ which we have translated as either “-hat” or “-his”, and we have the original word “jdq1f9″ which we have translated as either “tha–s” or “whi–t”.  That second one seems like a good candidate.  Now I need a word list. I will use a classic English word list from Donald E. Knuth.  You must use a word list appropriate for the cleartext you expect to find.  This would not help me for French text, nor would it help for government or corporate information which might contain many acronyms.

I have two possible six letter words identified: “tha–s” and “whi–t”.  Check the word list for words that match each pattern.  The following Perl command line will do it, assuming you have a word list file named ‘wordlist.txt’.

$ perl -ne 'print if m/^tha[a-z]{2}s$/' wordlist.txt
thanks
$ perl -ne 'print if m/^whi[a-z]{2}t$/' wordlist.txt
whilst

So only one word fits for each possibility.  I will go out on a limb and assume he used the word “Thanks” rather than “Whilst”.  I follow him on Twitter and I’ve seen him say “thanks”, but never “whilst”. Speakers of American English simply don’t use “whilst” very often.  Let’s take a look at the text if we assume the word “jdq1f9″ means “Thanks”.  We get two more letters, ‘1’=’n’ and ‘f’=’k’.

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'

THIS IS 4Hs Ssz2Nx z2xs z2N4sS4.  uH25vx -25 es4 4HIS 8IeH4 ANx zAN 
stAIv ts A4 Sz8I6sNvKINe[etAIv.z2t wI8S4@ -25 7Ivv es4 A }+& eIw4 
zA8x 42 GtA=2N.z2t.  THANKS w28 3vA-INe.

Looking better here.  Three words possibly done and nothing else looks too wrong.  I want to get that third word, after “This is”.  So what three letter words match the pattern “-h-“?

$ perl -ne 'print if m/^[a-z]h[a-z]$/i' wordlist.txt
aha
chi
ohm
oho
phi
rho
she
shh
shy
the
tho
thy
who
why

Which of those words make sense in a sentence following “This is”?  Only “the”, “who” and “why”.  I lean towards “who” and “why”, but if a capital ‘T’ at the beginning of a sentence has a different symbol from a lowercase ‘t’ in the middle of the sentence, “the” may do it.  This gives me a few more combinations to test:

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'

"This is who":

'4' = 'W'
's' = 'O'

THIS IS WHO SOz2Nx z2xO z2NWOSW.  uH25vx -25 eOW WHIS 8IeHW ANx zAN 
OtAIv tO AW Sz8I6ONvKINe[etAIv.z2t wI8SW@ -25 7Ivv eOW A }+& eIwW 
zA8x W2 GtA=2N.z2t.  THANKS w28 3vA-INe.

"This is why":

'4' = 'W'
's' = 'Y'

THIS IS WHY SYz2Nx z2xY z2NWYSW.  uH25vx -25 eYW WHIS 8IeHW ANx zAN 
YtAIv tY AW Sz8I6YNvKINe[etAIv.z2t wI8SW@ -25 7Ivv eYW A }+& eIwW 
zA8x W2 GtA=2N.z2t.  THANKS w28 3vA-INe.

"This is the":

'4' = 'T' (lowercase t!)
's' = 'E'

THIS IS THE SEz2Nx z2xE z2NTEST.  uH25vx -25 eET THIS 8IeHT ANx zAN 
EtAIv tE AT Sz8I6ENvKINe[etAIv.z2t wI8ST@ -25 7Ivv eET A }+& eIwT 
zA8x T2 GtA=2N.z2t.  THANKS w28 3vA-INe.

The last one gives me words 3 (“the”), 10 (“this”) and 16 (“at”).  My word list does not contain “whis” so I will throw out the two previous tries and continue from here.

Word 6 (“z214s94″) looks interesting with the pattern “–ntest”.  Only one word fits that pattern: “contest”.  It doesn’t surprise me one bit to find the word “contest” in the cleartext.  Assign ‘z’=’C’ and ‘2’=’O’.

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'
'4' = 'T'
's' = 'E'

Add in "contest":

'z' = 'C'
'2' = 'O'

THIS IS THE SECONx COxE CONTEST.  uHO5vx -O5 eET THIS 8IeHT ANx CAN 
EtAIv tE AT SC8I6ENvKINe[etAIv.COt wI8ST@ -O5 7Ivv eET A }+& eIwT 
CA8x TO GtA=ON.COt.  THANKS wO8 3vA-INe.

That gave me words 5 (“contest”), 13 (“can”) and 26 (“to”).  Now I want to take a look at the last word, “3vq-c1e” which I so far have matching the pattern “–a-in-“. The final encrypted ‘e’ also serves as the first letter in the three letter word “-et” (word 21).

53 words match the “–a-in-” pattern.  Of those 53, 44 of them (83%) end with “ing”, and would yield “get” for word 21.  I’ll take a leap here and assign ‘e’=’G’.

Looking at that first sentence, if ‘x’=’D’ then “This is the second code contest.” That makes perfect sense.

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'
'4' = 'T'
's' = 'E'
'z' = 'C'
'2' = 'O'

Add our new letters:

'e' = 'G'
'x' = 'D'

THIS IS THE SECOND CODE CONTEST.  uHO5vD -O5 GET THIS 8IGHT AND CAN 
EtAIv tE AT SC8I6ENvKING[GtAIv.COt wI8ST@ -O5 7Ivv GET A }+& GIwT 
CA8D TO GtA=ON.COt.  THANKS wO8 3vA-ING.

That gives me words 4 (“second”), 5 (“code”), 9 (“get”), 12 (“and”), 21 (“get”).

Word 17 (“9z8c6s1vfc1e[etqcv!z2t”), or “sc-i-en-king-g-ai-.co-” looks suspiciously like our host’s email address that he provided in the contest description.  Let’s substitute the letters to complete that.

Using the previous substitutions:

')' = ' '
'!' = '.'
'd' = 'h'
'j' = 'T'
'c' = 'I'
'9' = 'S'
'q' = 'A'
'1' = 'N'
'f' = 'K'
'4' = 'T'
's' = 'E'
'z' = 'C'
'2' = 'O'
'e' = 'G'
'x' = 'D'

Completing his email address:

'8' = 'R'
'6' = 'V'
'v' = 'L'
'[' = '@'
't' = 'M'

THIS IS THE SECOND CODE CONTEST.  uHO5LD -O5 GET THIS RIGHT AND CAN 
EMAIL ME AT SCRIVENLKING@GMAIL.COM wIRST@ -O5 7ILL GET A }+& GIwT 
CARD TO GMA=ON.COM.  THANKS wOR 3LA-ING.

Definitely on the right track here.  I can feel that Amazon.com gift card for $25.  The message even seems to mention it “-ill get a — gi-t card to -ma-on.com”.  From here, one only needs to plug in the letters and symbols that make sense and finish stepping through the process.

Filed under: Security

Oracle Enterprise Manager 12cR4 introduces the new “AWR Warehouse (AWRW)” feature, permitting administrators to consolidate AWR statistics from many individual databases managed by OEM into a single AWRW repository database.

As with all Oracle features, you must pay attention to licensing here.  I will not discuss licensing other than to point you to the relevant documents which you must read and understand yourself: Oracle Database Licensing Information 12c Release 1.

Documentation

At the moment official documentation appears limited to MOS note 1907335.1 and one section of the Oracle Database 2-Day Database + Performance Tuning Guide.  Get familiar with it.

Prerequisites

Repository Database

You must use Enterprise Edition for the AWRW repository database.  You must use version 12.1.0.2 or higher, or version 11.2.0.4 with patch 18547891 applied. Oracle recommends you use a database not used for any other purpose. I strongly agree with that recommendation.  Do not use your OEM repository. Note that I had to enable the diagnostic and tuning packs on the AWRW repository database by setting the control_management_pack_access initialization parameter to “DIAG+TUNING” before EM12c would allow me to select it for the repository.  I cannot reiterate enough how much I wish Oracle would explicitly state that users may enable management packs on their limited-use repository databases that support EM12c, RMAN catalogs and AWRW, but only a sucker expects license clarity out of Oracle.

I have selected 11.2.0.4 with patch 18547891 for my AWRW repository.

Oracle Enterprise Manager

You must use Oracle Enterprise Manager 12cR4 (12.1.0.4), and your OMS must have at least the August 31, 2014 bundle patch (19391521, or a later bundle patch) applied.  Your agents must run version 12.1.0.4.2 or later (requiring patch 19051570).

Licensing

Double check your licensing one more time.  Do not use features you have not licensed or you will pay a lot of money once you get audited, and you will get audited.

Configuration

For the purposes of this post I will skip the database installation and configuration steps.  If you have not yet gained proficiency with base installation and configuration tasks, you should probably gain some experience there before diving in to the AWR Warehouse.  Install a database of the appropriate version and register it with EM12c.

Planning

Think about your architecture.  With the recent release of AWRW functionality, some rough edges still exist.  These will probably get cleaned up over the next few releases but they took me by surprise and I have not seen them documented anywhere.

Oracle Enterprise Manager Agent Considerations

Do you use a separate dedicated user account on your servers to run the OEM Agent?  I do. Your AGENT_INSTANCE_DIR will get used by AWRW as a place to hold Data Pump output containing each source database’s AWR data.  I had to make this directory group writable by the dba group.  You also need to make sure the volume where this directory resides has enough free space to hold AWR extracts, which end up quite large on a busy system.  You may need to add more space if you keep your agent on a dedicated filesystem, as I do.

Do you run multiple instances under isolated accounts that don’t share a group membership?  You will probably need to create a group they all share that can write to the AGENT_INSTANCE_DIR.

Preferred Credential Considerations

AWRW strongly depends on the preferred credentials set for a database instance by the user that adds the database to AWRW.  If you already heavily use preferred credentials and want to use a different preferred database login for AWRW extraction compared to your usual DBA activities, you may elect to create a dedicated EM12c administrator to maintain AWRW to avoid conflicts.

The AWRW extraction user in the target database must have the DBA role and must also have an explicit execute grant on package SYS.DBMS_SWRF_INTERNAL.  I have chosen to use the SYSTEM account, to match my other preferred credential usage, but a more secure setup would use an account dedicated to this task.

Space Considerations

Take a look at how much space AWR consumes in your SYSAUX tablespaces already.  Your AWRW repository will need at least this much space, multiplied by however long you plan to keep these AWR snapshots around.  This will get very large, very quickly.

Added 20140912: I highly recommend that you disable data file autogrowth on your AWRW repository database.  I experienced repeated hangs until I determined that my jobs continually got stuck when SYSTEM or SYSAUX nearly filled and they sat there waiting on data file operations I/O as the system failed to resize the data files or identify a deadlock.  Do not rely on data file autogrowth, at least when using an 11.2.0.4 AWRW repository.

Initialize The AWR Warehouse

To begin configuring the AWR Warehouse, you must login using an EM12c super administrator account, like SYSMAN.  Once logged in, go to the Databases target list screen.  Unfortunately for this purpose you must use the “Database Load Map” form of the screen and not the infinitely more useful “Deprecated Search List (with Metrics)” that I have up on screen 99.9% of the time. Click the Targets menu, select Databases from the submenu that appears, and then if you do not see a “Performance” menu, enable the “Database Load Map” radio button.

Click the Performance menu and select the “AWR Warehouse” item.

This button makes things happen

At this point, if you used a super administrator account, you should see a summary screen that briefly describes the feature with a link to click to begin configuration.  If you don’t, log out and come back with the SYSMAN account.

Click Configure to continue

The next screen offers a search box to select the database to use as your AWRW repository and the usual credential selector.  Select the correct database, choose a database credential (I first selected SYSTEM, which failed, so use SYS as SYSDBA) and provide host credentials.

Rough edge: no warning that you must use SYSDBA

Once you click Next, the tool will pop up a dialog box warning you to make sure that your repository database has the necessary patch installed, and then asks you to select how long the system should keep AWRW data.  You can also select a staging location for AWR data extract storage prior to data loading.

Diamonds and AWR Warehouses are forever

Click Submit on this screen and OEM will submit a job to initialize the AWRW repository.  To find this job later, if needed, go to the advanced job activity page and search for jobs of type “dbSetupCAW”.  The job should complete successfully if you have done everything correct so far.  On my system it only took six seconds, so just wait a moment and then reload the page, which should now look like this.

That was easy

Click on the database icon at the upper left to switch away from the repository configuration tab to the database selection tab.

No data yet

As of this point you no longer need to use the SYSMAN account.  I switched back to my regular account, then returned to this screen.

Click the Add button to begin adding your first database(s). OEM will prompt you with the usual target selection screen.  Choose one or more databases and then click the Select button.  AWRW will NOT prompt you for credentials at this time.  Instead it will silently use the database host and normal database user preferred credentials you have established for the database target.  Another rough edge I expect to work better in future versions.  AWRW will perform some initial validations of those credentials to make sure that the database user has the DBA role and the previously mentioned execute grant on SYS.DBMS_SWRF_INTERNAL.  If you have missed any of these steps OEM will tell you about it and prevent you from adding the database.  Again, later I expect this to include an automated setup that will fix those issues.

I can’t show you the name

At this point you can just walk away and within about 24 hours you should have AWR data loaded into the warehouse.  If you feel impatient, click on one of the lines for a database to select it, then choose “Upload Snapshots Now” from the Actions menu.  This will submit a job to extract and load the AWR data, which you can find later under the job type “Run AWR Extract, Transfer and Load”.  In the background, this job extracts AWR data to the AGENT_INSTANCE_DIR on the target database’s server, compresses the data, transfers it to the staging area defined during AWRW repository setup, then loads the transferred data into the consolidated AWR Warehouse repository.

One database in there. So many to go.

Summary

The size of and load on your selected database, along with the amount of AWR history you keep online, will influence how long each load takes.  My first load appeared to hang, with the AWRW repository database full of sessions waiting on enq: HW contention and buffer busy waits.  I ended up doing a shutdown abort and following the workaround instructions in MOS note 1912230.1.  I do not know if I truly needed to do this or not, but the symptoms sounded similar.  I’ve also noticed that some limits appear to exist.  I keep 42 days worth of hourly snapshots in each AWR, and my initial load only picked up 20 days / 500 snapshots.  This may represent rate-limiting as a way to trickle-load the AWRW, or it may mean AWRW does not yet play nicely with large AWR data.  Time will tell, and I fully expect future versions to shake out any bugs and to hold the DBA’s hand a bit more through the process.

I hope to cover using AWRW for performance tuning in a later post and I look forward to comments.

More Information

See these other fine posts for more information.

• AWR Warehouse in EM12c Rel. 4
• AWR Warehouse in EM12c, Rel 4, Part II
• AWR Warehouse Webinar from ODTUG
• AWR Warehouse, Status

With the recent POODLE vulnerability, server operators must now (finally) disable SSL version 3.0 and move up to TLS 1.0 at the minimum, if not TLS 1.2.

UPDATE: Many thanks to Courtney Llamas who provided me with a link to the section of the documentation that describes the right way to do this.  If you want to disable SSLv3 in EM12c, follow the instructions in section 2.3.2.4 of chapter 2 of the Oracle Enterprise Manager Cloud Control Security Guide.  You will need to re-secure your OMS during the process and this will require that you have access to the SYSMAN password and an agent registration password. I can confirm that the steps in this document work and do disable SSLv3. Make sure you follow the steps in the document to secure the management agents, too.

What follows demonstrates what happens when someone with a little too much information doesn’t read through the documentation.  Don’t do what I did.  Do it the right way, as described above.

I do not wish to wait for Oracle to provide a procedure to disable SSL version 3.0 in Oracle Enterprise Manager 12c, so I have come up with this process.  This will apply to Oracle Enterprise Manager 12c, PS3, PSU1 (aka 12.1.0.4.1, aka EM12c R4).  This process may work on older versions.

IMPORTANT DISCLAIMER: Oracle Support has not blessed this process and should provide an official method of fixing this issue soon.  You take all responsibility for any issues you encounter by following these instructions.  Of course, you also take the responsibility for operating your EM12c system in a way by which your encrypted connections can be trivially hacked, so make a decision appropriate to your environment.  I recommend you do NOT follow these steps and instead raise a priority Sev-1 service request with Oracle demanding an immediate, supported fix, but for those in a situation requiring an immediate response, I offer this post.

Do I have SSL v3 enabled now?

If you have not made a specific effort to disable it or applied a future patchset in which Oracle disables the functionality, you have SSL v3 enabled.

How do I test for SSL v3 in EM12c?

How can you test and how can you validate that you have disabled it once complete?  I advise that you open up a copy of the Firefox browser, and go to the about:config page.  In about:config, search for “security.tls.version”.  You can (as of the date I wrote this post) force Firefox to use only SSLv3 by setting both security.tls.version.min and security.tls.version.max to the value 0.  The image below shows a secure setting of 1, and 3 (for TLS 1.0 and TLS 1.2 respectively).

Double-click on each of the top two lines and change the setting to 0.  The screen should now look like the image below.

Make sure you change this setting back once you have finished testing or you have created a huge security risk for this browser.  See my previous post on Firefox security for more information.

Now that you have forced Firefox to use only SSLv3, navigate to your Oracle Enterprise Manager login screen.  If the page loads and you see a login prompt like below, your EM12c installation allows insecure SSLv3 connections.

After disabling SSLv3 you should return to this section, repeat these steps, and make sure that you get an error page instead of the login screen.

Disabling SSLv3 in EM12c Oracle HTTP Server

As an aside, MOS note 1935500.1 appears to provide the master note for details about mitigating the POODLE attack in Oracle products.  In this instance, we specifically care about disabling SSLv3 in the Oracle HTTP Server (OHS) used by Oracle Enterprise Manager, and the documentation to make this change is available to the public in the Oracle HTTP Server documentation.

Specifically, you need to modify the SSLProtocol declaration in four different configuration files:

$EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/httpd_em.conf
$EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/ssl.conf
$MW_HOME/Oracle_WT/ohs/conf/ssl.conf
$OMS_HOME/sysman/config/httpd_em.conf.template

In the first three files, find the line that looks like:

 SSLProtocol all -SSLv2

Change this line to:

SSLProtocol all -SSLv2 -SSLv3

In the last file, find the line that looks like:

SSLProtocol &PROTOCOL%

Change this line to:

SSLProtocol all -SSLv2 -SSLv3

Then bounce your entire EM12c stack and return to the test procedure at the top of this post, force Firefox to SSLv3, connect to Oracle Enterprise Manager.  You should now see an error like the following:

Success!  Now revert those Firefox security.tls.version.min and security.tls.version.max settings back to 1 and 3 respectively, and return to EM12c and you should see the login page again.

Congratulations, you’ve just turned off SSLv3 in EM12c.  Please save these steps so that you can revert them back to the out-of-the-box behavior if requested to do so by Oracle or when they provide an official fix.  I hope to update this post in the future with similar instructions for the Weblogic admin console but I really hope Oracle Support beats me to it.

You probably will not encounter this issue, but I will post this anyway to get the error message and resolution indexed by Google.

While attempting to apply patch 19513382 (EM agent bundle patch 12.1.0.4.3) to my EM12cR4 agents, I ran into multiple problems.  Initially it would not apply to any of my agents.  Bug 20134182 and the resolution described in MOS note 1952355.1 resolved the first problem (OPatch reporting that identical patches 18721761 and 18502187 already exist), but that left me with one agent I could not upgrade. Attempts to run patch plan validation within EM12c produced the following error:

PatchList : 19513382
PatchLocList : /tmp/p19513382_600000000009641_2000_0/oraagent
TargetName : [redacted]:[port]
----------------------------------------
[11_12_2014_10_00_40] Command Arguments:
/oraagent/agent12c/core/12.1.0.4.0/OPatch/opatch checkComponents -phbasedir /tmp/p19513382_600000000009641_2000_0/oraagent/19513382 -oh /oraagent/agent12c/core/12.1.0.4.0 -invPtrloc /oraagent/agent12c/core/12.1.0.4.0/oraInst.loc

OPatch cannot continue because it would not be able to load OUI platform dependent library from the directory "/oraagent/agent12c/core/12.1.0.4.0/oui/lib/linux64". The directory does not exist in the Oracle home.
This could be due to the following reasons.

(1) Incompatible usage of java with OUI (32/64 bit).
(2) Wrong 32-bit Oracle Home installation in 64-bit environment (or) vice-versa.
Please contact Oracle support for more details.

OPatch failed with error code 1

PREREQ_CONTEXT_HOST_NAME:[redacted]
REREQ_CONTEXT_HOME_LOCATION:/oraagent/agent12c/core/12.1.0.4.0
PREREQ_NAME: Checking if the patches are applicable.
PREREQ_DESC: Checking if the patches are applicable on the Management Agent.
PREREQ_TYPE:APPLICABILITY
PREREQ_STATUS:FAILED
PREREQ_MESG: None of these patches are applicable on the Management Agent.
PREREQ_MESG_PATCH:19513382
PREREQ_REMEDY:MANUAL
PREREQ_REMEDY_DETAILS: Remove patch(es) 19513382 from this patch plan.

I already know from the previously referenced MOS note that OPatch 11.1.0.12.3 contains bugs, so as a first debugging step I attempted to rollback the OPatch upgrade by restoring the backup copies of OPatch found in $AGENT_HOME/OPatch/backup/.  I received the same error message with OPatch 11.1.0.10.4 and 11.1.0.11.0.  I also received a similar error even if I simply tried to run “opatch lsinv” from the command line with the older versions. So OPatch did not cause this issue.

Since the error message mentions 32-bit and 64-bit incompatibility, I needed to consider the environment.  This server runs Linux x86-64 (SLES 10 SP4), but must use a 32-bit EM agent based on the certification matrix and MOS note 1488161.1. I next checked to find my last successful patch run, which happened only a month ago, so a recent change has to have caused this problem. Going through my notes, the only recent change on this server involved upgrading the JDK used by the EM agent per MOS note 1944044.1.

Luckily I still had the old JDK available for comparison.

> java -version
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) Server VM (build 20.14-b01, mixed mode)
> file `which java`
/oraagent/agent12c/core/12.1.0.4.0/jdk/bin/java: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Looking at the new JDK:

> ./java -version
java version "1.6.0_85"
Java(TM) SE Runtime Environment (build 1.6.0_85-b13)
Java HotSpot(TM) 64-Bit Server VM (build 20.85-b01, mixed mode)
> file java
java: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

There I have my problem. In upgrading the JDK, I had installed the 64-bit version of Java 1.6u85, not the 32-bit version, based on the fact that the server runs a 64-bit OS. I had not considered that a 64-bit JDK would not remain compatible with the 32-bit agent on this 64-bit system.

Surprisingly, everything about the agent seems to have worked fine, despite the 64-bit JDK.  Nothing broke until I attempted to use OPatch.

To resolve the issue, I stopped the agent and moved the original 32 bit 1.6u43 JDK back to where it belongs, followed note 1952355.1 to work around the known bugs when using OPatch 11.1.0.12.3 to apply 19513382, then successfully applied the patch.  After that I downloaded the correct 32-bit version of the 1.6u85 JDK, installed it per 1944044.1, and now OPatch works as expected.

By default, when an administrator configures Oracle Enterprise Manager 12c to use SSL, the system will use a default self-signed certificate, provided for demo purposes only.  The documentation states repeatedly that users should not use these certificates in a production environment, as they represent a security risk. This blog post documents, step by step, a process to replace these demo certificates with custom third party certificates, across the OMS console, OMS upload port, agents, and WebLogic Server. I will follow this process on a single-OMS configuration; if you have more than one OMS please consult the documentation for more details, as your process will vary and the steps I have provided may break your system.

I have tested these instructions on Linux x86-64 (SLES11 SP3) with EM12c R4 PSU2 (12.1.0.4).

Official Documentation

The official documentation for this process resides in the following My Oracle Support notes:

• Using ORAPKI Utility to Create a Wallet with Third Party Trusted Certificate and Import into OMS (Doc ID 1367988.1)
• EM 12c Cloud Control How to Create a Wallet With Third Party Trusted Certificate that Can Be Imported into the OMS For SSL Comunication ? (Doc ID 1399293.1)
• 12c Cloud Control: Steps to Import Third Party Trusted SSL Certificate into 12c Cloud Control Agent URL (Doc ID 1593183.1)
• 12c Cloud Control: Steps to Create and Import Third Party / Self-Signed SSL Certificates for WebLogic Server in an Enterprise Manager Installation (Doc ID 1527874.1)
• How to Create a Java Keystore via Keytool in FMW 11g/12c (Doc ID 1230333.1)

Why Should I Do This?

You may not fully understand the mechanics of SSL/TLS certificates and the chain of trust. I cannot fully explain this complex topic in a blog post, but if you need a reason to make this change other than demands from your organizational security/compliance team, please take Oracle’s word for it, and notice this text that appears in your GCDomain.log file when you run your system with the provided default demo certificates:

####<Feb 12, 2015 2:58:07 PM EST> <Alert> <Security> <[hostname redacted]> <EMGC_ADMINSERVER> <VDE Replication Thread> <<WLS Kernel>> <> <> <1423771087209> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
[
 Version: V3
 Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
 Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4

 Key: Sun RSA public key, 512 bits
 modulus: 9550192877869244258838480703390456015046425375252278279190673063544122510925482179963329236052146047356415957587628011282484772458983977898996276815440753
 public exponent: 65537
 Validity: [From: Thu Mar 21 15:12:27 EST 2002,
 To: Tue Mar 22 16:12:27 EDT 2022]
 Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
 SerialNumber: [ 33f10648 fcde0deb 4199921f d64537f4]

Certificate Extensions: 1
[1]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
 Key_CertSign
]

]
 Algorithm: [MD5withRSA]
 Signature:
0000: 9D 26 4C 29 C8 91 C3 A7 06 C3 24 6F AE B4 F8 82 .&L)......$o....
0010: 80 4D AA CB 7C 79 46 84 81 C4 66 95 F4 1E D8 C4 .M...yF...f.....
0020: E9 B7 D9 7C E2 23 33 A4 B7 21 E0 AA 54 2B 4A FF .....#3..!..T+J.
0030: CB 21 20 88 81 21 DB AC 90 54 D8 7D 79 63 23 3C .! ..!...T..yc#<

] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>

Read that again if you didn’t catch it the first time through: “The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.” This text comes from code in WebLogic, not from me. Here Oracle tells you very explicitly that your system currently contains a severe vulnerability.

You will also notice that when using the EM12c console, or accessing an agent URL, or accessing the WebLogic Server administration console may show warnings in your browser about untrusted certificates. Once you replace your certificates as described in the documentation above or my steps below, you will no longer have those issues.

Using 3rd Party SSL/TLS Certificates With EM12c

Overview

You will follow 7 high level steps to complete the process of securing your EM12c environment with custom third party SSL/TLS certificates.

1. Create an Oracle wallet for the OMS.
2. Secure the OMS console using the OMS wallet.
3. Secure the OMS upload port using the OMS wallet.
4. Re-secure all agents.
5. Create Oracle wallets for agents.
6. Configure the agents to use their wallets.
7. Secure WebLogic with the OMS wallet.

Create an Oracle wallet for the OMS

First we follow steps 1a through 1h from document 1367988.1.  All these steps occur on the OMS host.

Disable shell history (optional but recommended)

While following these steps, you will repeatedly have to type passphrases on the command line. To avoid having these passphrases stored in your Oracle user’s shell history, disable history saving.  In the bash shell that I use, I accomplish this by unsetting the HISTFILE variable. You may need to use another mechanism in another shell.

$ unset HISTFILE

Use the correct ORAPKI command

You should use the ORAPKI command from your middleware home’s oracle_common/bin directory.  I will refer to this as $MW_HOME/oracle_common/bin/orapki in the following instructions.

Create an Oracle wallet

The documentation specified that we should create an auto-login wallet, but in my single-OMS setup, I believe that I will achieve better security with an auto-login-local wallet, as the auto-login feature will only function on this specific host. You will need to select a base directory for your OMS wallet.  I used $ORACLE_BASE/oemwallet. ORAPKI will prompt you for a password. Use a secure one, and note it down somewhere safe. You will use it many times during this process.

$ mkdir $ORACLE_BASE/oemwallet
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/oemwallet -auto_login_local
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Get in the habit of displaying the wallet contents after each operation to confirm that everything worked.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Create a key within the wallet.  Make sure you replace omshost.domain.com with the fully qualified domain name of your OMS host. I highly recommend using a 2048 bit keysize, as shown below. Include the wallet password you specified earlier on the commandline as the -pwd argument, contained in single quotes. Display the wallet again afterward.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 Subject: CN=omshost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 User Certificates:
 Trusted Certificates:
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US

Export a certificate signing request based on this key. Make sure the -dn you specify exactly matches the -dn specified earlier. Provide a filename in the -request argument in which to store the certificate signing request (CSR).

$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/oemwallet -dn "CN=omshost.domain.com, OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/EM12cCSR.txt
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Submit this CSR file to your signing authority. Inform them that you MUST have a single-host certificate with your OMS host’s fully qualified domain name in the CN field. Subject Alternate Name (SAN) certificates or wildcard certificates will not work at all. Your signing authority should then provide you with a root certificate, an intermediate certificate, and a user certificate.

Import the root, intermediate, and user certificates into the OMS wallet. Note that you must import the root and intermediate certificates using -trusted_cert, and the user certificate using -user_cert.  I used DigiCert, and I can confirm that their certificates function correctly in EM12c and recommend their service.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/TrustedRoot.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -trusted_cert -cert ~/DigiCertCA2.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/oemwallet -user_cert -cert ~/omshost.domain.com.cer -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Display the wallet contents after this operation.

$  $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/oemwallet
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=omshost.domain.com,OU=[REDACTED],O=[REDACTED],L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Concatenate the root and intermediate certificates into a single file, which you will use later.

$  cat ~/DigiCertCA2.cer ~/TrustedRoot.cer > $ORACLE_BASE/trusted_certs.txt

You have completed configuration of your OMS wallet.

Secure the OMS console

Now, using emctl from the $OMS_HOME, tell EM12c to secure the OMS console using the certificate contained in your wallet. The system will prompt you for the SYSMAN password and inform you to restart the entire OMS once complete.

$ $OMS_HOME/bin/emctl secure console -wallet /oracle/oem/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing Console... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing Console... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Now access your OMS console with your favorite browser and confirm that your new certificate appears.  Your certificate should show a trusted path back to a root certificate, and your browser should produce no warnings.

At this point, you have secured communication between your browser and the EM12c OMS console with your custom certificate.  You still have more work to do though. Your agents upload monitoring data to the OMS upload port, and it still uses the demo certificate. Fix that in the next step.

Secure the OMS upload port

Secure the OMS upload port. Expect to receive email or pager alerts after this step, as once you restart the OMS, none of your agents can communicate with it, as they expect to see the demo certificates on the upload port. You will need to provide the SYSMAN password as well as an agent registration password.

$ $OMS_HOME/bin/emctl secure oms -wallet $ORACLE_BASE/oemwallet -trust_certs_loc $ORACLE_BASE/trusted_certs.txt
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing OMS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Enter Agent Registration Password :
 Securing OMS... Successful
 Restart OMS
$ $OMS_HOME/bin/emctl stop oms -all ; sleep 5 ; $OMS_HOME/bin/emctl start oms

Re-secure all agents

Now you must re-secure all of your agents so that they can resume uploading data to the OMS console and monitoring your systems. Execute the following steps on every agent, using emctl from the agent home.  You will need to provide an agent registration password to complete this process.

$ $AGENT_HOME/bin/emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Agent successfully stopped... Done.
Securing agent... Started.
Enter Agent Registration Password :
Agent successfully restarted... Done.
Securing agent... Successful.
$ $AGENT_HOME/bin/emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

It may take a little while for the OMS to process the new agents and their uploads, but once you have run this process on every agent they should all communicate successfully with the OMS and appear as OK from the agent management screen.

Create Oracle wallets for agents

Next we secure the agent URLs. The OMS connects to the agents at this URL to submit management requests. At the moment, the agents still use self-signed certificates to secure this URL.  For this process we create an Oracle wallet, on the OMS host, using the same ORAPKI command as for the OMS wallet. We will generate a certificate signing request from each agent wallet, submit those CSRs to a certificate authority, and import the received certificates.

As with the OMS, the agents must use single-host certificates, not wildcard or subject alternate name (SAN) certificates.  To determine the correct fully qualified domain name for each agent, execute emctl status agent from the agent home.

$  $AGENT_HOME/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /oraagent/agent12c/agent_inst
Agent Log Directory : /oraagent/agent12c/agent_inst/sysman/log
Agent Binaries : /oraagent/agent12c/core/12.1.0.4.0
Agent Process ID : 12480
Parent Process ID : 12359
Agent URL : https://agenthost.domain.com:3872/emd/main/    <---- use this hostname
Local Agent URL in NAT : https://agenthost.domain.com:3872/emd/main/
Repository URL : https://omshost.domain.com:4902/empbs/upload
Started at : 2015-03-03 12:41:45
Started by user : oraagent
Operating System : Linux version 3.0.101-0.40-default (amd64)
Last Reload : (none)
Last successful upload : 2015-03-03 13:41:48
Last attempted upload : 2015-03-03 13:41:48
Total Megabytes of XML files uploaded so far : 0.4
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 56.64%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2015-03-03 13:48:56
Last successful heartbeat to OMS : 2015-03-03 13:48:56
Next scheduled heartbeat to OMS : 2015-03-03 13:49:56

---------------------------------------------------------------
Agent is Running and Ready

Repeat these steps for every agent.

Create a directory to store the agent wallet, and an agent wallet. This time do NOT use -auto_login_local, use only -auto_login, as you will distribute these wallets to the agent hosts after generating them on the OMS host.  Use a strong password, and save it for later, as you will reuse it many times.

$ mkdir $ORACLE_BASE/agentwallets
$ mkdir $ORACLE_BASE/agentwallets/agenthost.domain.com
$ $MW_HOME/oracle_common/bin/orapki wallet create -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -auto_login
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Enter password:
 Enter password again:

Create the certificate, then a certificate signing request, saving it to file ~/agenthost.domain.com.csr. Again, I recommend a 2048 bit certificate.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -keysize 2048 -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet export -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -dn "CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US" -request ~/agenthost.domain.com.csr
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

As before, submit this certificate signing request to your certificate authority, and receive back three files containing a root certificate, an intermediate certificate, and a user certificate. Import these into the agent wallet, and display the wallet afterwards to confirm everything imported successfully.

$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/TrustedRoot.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -trusted_cert -cert ~/DigiCertCA.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet add -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com -user_cert -cert ~/agenthost.domain.com.crt -pwd '[REDACTED]'
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.
$ $MW_HOME/oracle_common/bin/orapki wallet display -wallet $ORACLE_BASE/agentwallets/agenthost.domain.com
 Oracle PKI Tool : Version 11.1.1.7.0
 Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

 Requested Certificates:
 User Certificates:
 Subject: CN=agenthost.domain.com,OU=EM,O=Organization,L=City,ST=State,C=US
 Trusted Certificates:
 Subject: CN=DigiCert High Assurance EV Root CA,OU=www.digicert.com,O=DigiCert Inc,C=US
 Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
 Subject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
 Subject: CN=DigiCert High Assurance CA-3,OU=www.digicert.com,O=DigiCert Inc,C=US

You have finished creating this agent’s wallet.  Repeat this for every agent.

Configure the agents to use their wallets

Inside the agent wallets you’ve just created, you will find a cwallet.sso file. Take this file from each agent’s wallet and copy it to the agent server. Stop the agent, then place the file into $AGENT_INSTANCE_DIR/sysman/config/server/ and set the permissions to 640, then start the agent.

$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Stopping agent ..... stopped.
$ cp cwallet.sso $AGENT_INSTANCE_DIR/sysman/config/server
$ chmod 640 $AGENT_INSTANCE_DIR/sysman/config/server
$ $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............. started.

Next, visit the agent URL in your favorite web browser and examine the certificate it uses.  You should now see that it uses the 3rd party SSL/TLS certificate that you installed.

Secure WebLogic with the OMS wallet

Now the OMS (both console and upload ports) and agents will use your new certificates. This leaves WebLogic as the one remaining component needing your new certificates. Please note in following the below directions that securing WebLogic with a wallet only works as of EM12c R3, earlier versions must use a Java keystore. See note 1527874.1 for more information.

First import the root and intermediate certificates to the keystore on the OMS host’s agent. Use the default password welcome for the agent keystore, and alias names rootcacert and intercacert.

$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/TrustedRoot. crt -alias rootcacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS
$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc ~/DigiCertCA. crt -alias intercacert -password welcome
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.

 Message : Certificate was added to keystore
 ExitStatus: SUCCESS

Back up some WLS configuration files, just in case, before securing WLS with your certificate.  If you have problems in this step, make sure you have stopped all WLS processes, then restore these files from backup.

$ mkdir ~/wlscertbak
$ cp -a $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/NodeManager/emnodemanager/nodemanager.properties ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/keystores/proxy ~/wlscertbak/
$ cp -a $EM_INSTANCE_BASE/user_projects/domains/GCDomain/config/config.xml ~/wlscertbak/

Stop the OMS.

$ $OMS_HOME/bin/emctl stop oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Successfully Stopped
 Oracle Management Server is Down

Secure WLS using the OMS wallet created earlier. You will need to provide the SYSMAN password.

$ $OMS_HOME/bin/emctl secure wls -wallet $ORACLE_BASE/oemwallet
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Securing WLS... Started.
 Enter Enterprise Manager Root (SYSMAN) Password :
 Securing WLS... Successful
 Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
 If there are multiple OMSs in this environment, perform this configuration on all of them.

Stop the entire WLS stack, then start the OMS.

$ $OMS_HOME/bin/emctl stop oms -all
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Already Stopped
 Oracle Management Server is Down
 Stopping BI Publisher Server...
 BI Publisher Server Successfully Stopped
 AdminServer Successfully Stopped
 BI Publisher Server is Down
$ $OMS_HOME/bin/emctl start oms
 Oracle Enterprise Manager Cloud Control 12c Release 4
 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
 Starting Oracle Management Server...
 Starting WebTier...
 WebTier Successfully Started
 Oracle Management Server Successfully Started
 Oracle Management Server is Up
 Starting BI Publisher Server ...
 BI Publisher Server Successfully Started
 BI Publisher Server is Up

Visit the WebLogic Server administration console and you should now see that it presents your custom SSL/TLS certificate and no longer uses the demo certificate.

Conclusion

If you have successfully followed these steps, your system should now use your custom SSL/TLS certificates everywhere, and you should no longer experience certificate warnings in your browsers.

See Also

• EM12c R4 SSL Security Checkup Script

[LATEST SCRIPT UPDATE: 20150501, VERSION 1.3, covers 20150430 patch release]

With all the recent news on companies getting hacked and attacks on encryption techniques, you need to act proactively to secure your Oracle Enterprise Manager Cloud Control 12c environment. Do not wait for your employer’s auditor to come around and send you a report of all the flaws in your system.

To put it in very simple terms, if you do not do the following across EVERY EM12c component, you should consider your setup vulnerable:

• Disable SSLv2 and SSLv3
• Enable TLSv1
• Disable weak ciphersuites such as those using the MD5 or RC4 algorithms, or those previously designed for export outside the USA back in the 1990s, or those that do not use enough key bits for encryption.
• Eliminate the use of self-signed certificates, especially those requiring the user to trust the demo WebLogic certificate authority.
• Stay current on EM12c base releases (currently EM12c R4)
• Stay current on PSU updates to EM12c (PSU3 as of April 2015)
• Stay current on monthly system patch bundles
• Stay current on quarterly critical patch update alerts for all EM12c components – note that you have to pay attention to, for example, Oracle HTTP Server (OHS) critical patch updates, as EM12c distributes and relies on OHS. See MOS note 1664074.1 for a good, but incomplete list of patches needed.
• Stay current on repository database patch set updates
• Stay current on EM12c Java versions [EDIT: 20150415: Added Java check to script]

Yes, this takes a lot of work.  Yes, we do not have great documentation to assist in the process.  Yes, you can contact Oracle support for assistance.

Yes, you do need to deal with EVERY endpoint for the SSL configuration.  That includes:

• OMS console
• OMS upload port
• OMS console proxy port
• Management agents
• EM Node Manager
• WebLogic Server administration console
• OHS administration port
• OPMN port
• BI Publisher

In the meantime, though, you need to have a good idea of where your system has flaws so that you know where to spend your time fixing it. To help with this, I have created a script that will examine your EM12c environment, find all the ports in use, check for SSLv2, SSLv3, and TLSv1, validate the cipher suites in use, check to make sure you have current patches installed, check for the usage of self-signed certificates on SSL/TLS endpoints, and check for current Java JDK versions in EM12c components. [EDIT: 20150311: Added self-signed certificate check]. [EDIT: 20150313: Added patch check for repository databases on same host as OMS server. I have only tested this on an 11.2.0.4 repository, but I believe it will work for the 12.1.0.2 repository just recently re-certified. If it fails for you please let me know.] [EDIT: 20150415: Added check for Java JDK versions.]

This script does not require any arguments or configuration. I have tested it ONLY on EM12c R4 and on Linux x86-64 and only on single-host OMS environments.  It may or may not work on other operating systems.  To run this script, copy it from the end of this post (or from the pastebin link here [EDIT: 20150501: updated link to v1.3], and execute it as the Oracle software owner on your OMS host, with your environment fully up and running. [EDIT: 20150311: Updated script incorporating feedback from Dave Corsar and opa tropa to support Solaris and AIX.]

The script will not make any changes to your system.  Mostly it crawls your configuration files to identify ports, then tests them with the openssl s_client command and various command line arguments to identify protocol and cipher suite usage, and whether or not it can find self-signed certificates.  At the end it runs OPatch checks for current needed security and functionality patches.

As of the version 1.1 release, I will mark newly checked patches with “*NEW*” in the script output and updated patches with “*UPDATED*”. For example, when a new PSU patch comes out, I will mark it as an update, but I will mark new (or previously not checked) patches as new. [EDIT: 20150415: This paragraph added.]

Example output from my fully patched and secured system [EDIT: 20150311: Unfortunately I still have self-signed certificates for OPMN and the OHS administration port, so my sample output now includes some failed checks]:

Performing EM12cR4 security checkup version 1.3.

Using port definitions from configuration files
    /etc/oragchomelist
    /oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
    /oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties
    /oracle/oem/gc_inst1/WebTierIH1/config/OPMN/opmn/ports.prop
    /oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/admin.conf

    Agent port found at omshost.domain.com:3872
    BIPublisher port found at omshost.domain.com:9702
    NodeManager port found at omshost.domain.com:7404
    OHSadmin port found at omshost.domain.com:9999
    OMSconsole port found at omshost.domain.com:7803
    OMSproxy port found at omshost.domain.com:7302
    OMSupload port found at omshost.domain.com:4902
    OPMN port found at omshost.domain.com:6701
    WLSadmin found at omshost.domain.com:7103

    Repository DB version=11.2.0.4.0 SID=emrep host=omshost.domain.com
    Repository DB on OMS server, will check patches in /oracle/oem/product/11.2.0/dbhome_2

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

    (1a) Forbid SSLv2 connections
    Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
    Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9702... OK
    Confirming ssl2 disabled for NodeManager at omshost.domain.com:7404... OK
    Confirming ssl2 disabled for OHSadmin at omshost.domain.com:9999... OK
    Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7803... OK
    Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7302... OK
    Confirming ssl2 disabled for OMSupload at omshost.domain.com:4902... OK
    Confirming ssl2 disabled for OPMN at omshost.domain.com:6701... OK
    Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7103... OK

    (1b) Forbid SSLv3 connections
    Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
    Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9702... OK
    Confirming ssl3 disabled for NodeManager at omshost.domain.com:7404... OK
    Confirming ssl3 disabled for OHSadmin at omshost.domain.com:9999... OK
    Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7803... OK
    Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7302... OK
    Confirming ssl3 disabled for OMSupload at omshost.domain.com:4902... OK
    Confirming ssl3 disabled for OPMN at omshost.domain.com:6701... OK
    Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7103... OK

    (1c) Permit TLSv1 connections
    Confirming tls1 available for Agent at omshost.domain.com:3872... OK
    Confirming tls1 available for BIPublisher at omshost.domain.com:9702... OK
    Confirming tls1 available for NodeManager at omshost.domain.com:7404... OK
    Confirming tls1 available for OHSadmin at omshost.domain.com:9999... OK
    Confirming tls1 available for OMSconsole at omshost.domain.com:7803... OK
    Confirming tls1 available for OMSproxy at omshost.domain.com:7302... OK
    Confirming tls1 available for OMSupload at omshost.domain.com:4902... OK
    Confirming tls1 available for OPMN at omshost.domain.com:6701... OK
    Confirming tls1 available for WLSadmin at omshost.domain.com:7103... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)
    Checking LOW strength ciphers on Agent (omshost.domain.com:3872)...    OK
    Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)...    OK
    Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)...    OK

    Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9702)...    OK
    Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9702)...    OK
    Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9702)...    OK

    Checking LOW strength ciphers on NodeManager (omshost.domain.com:7404)...    OK
    Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7404)...    OK
    Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7404)...    OK

    Checking LOW strength ciphers on OHSadmin (omshost.domain.com:9999)...    OK
    Checking MEDIUM strength ciphers on OHSadmin (omshost.domain.com:9999)...    OK
    Checking HIGH strength ciphers on OHSadmin (omshost.domain.com:9999)...    OK

    Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7803)...    OK
    Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7803)...    OK
    Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7803)...    OK

    Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7302)...    OK
    Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7302)...    OK
    Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7302)...    OK

    Checking LOW strength ciphers on OMSupload (omshost.domain.com:4902)...    OK
    Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4902)...    OK
    Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4902)...    OK

    Checking LOW strength ciphers on OPMN (omshost.domain.com:6701)...    OK
    Checking MEDIUM strength ciphers on OPMN (omshost.domain.com:6701)...    OK
    Checking HIGH strength ciphers on OPMN (omshost.domain.com:6701)...    OK

    Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7103)...    OK
    Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7103)...    OK
    Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7103)...    OK


(3) Checking self-signed certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
    Checking certificate at Agent (omshost.domain.com:3872)... OK
    Checking certificate at BIPublisher (omshost.domain.com:9702)... OK
    Checking certificate at NodeManager (omshost.domain.com:7404)... OK
    Checking certificate at OHSadmin (omshost.domain.com:9999)... FAILED - Found self-signed certificate
    Checking certificate at OMSconsole (omshost.domain.com:7803)... OK
    Checking certificate at OMSproxy (omshost.domain.com:7302)... OK
    Checking certificate at OMSupload (omshost.domain.com:4902)... OK
    Checking certificate at OPMN (omshost.domain.com:6701)... FAILED - Found self-signed certificate
    Checking certificate at WLSadmin (omshost.domain.com:7103)... OK

(4) Checking EM12c Oracle home patch levels against 30 April 2015 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

    (4a) OMS (/oracle/oem/Middleware12cR4/oms) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.3 PSU Patch (20392036)... OK
Patch 20392036 : applied on Wed Apr 15 09:00:16 EDT 2015 19362386, 20392036, 19452280, 19170861, 18356966, 19055979, 18353162

    (4b) BI Publisher (/oracle/oem/Middleware12cR4/Oracle_BI1) CPUJAN2015 Patch (19822893)... OK
19822893 19822893 Patch 19822893 : applied on Wed Feb 25 09:16:21 EST 2015

    (4b) BI Publisher (/oracle/oem/Middleware12cR4/Oracle_BI1) Merge Patch (20444447)... OK
Patch 20444447 : applied on Wed Feb 25 09:21:03 EST 2015

    (4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) CVE-2015-0426 Oracle Help Patch (20075252)... OK
Patch 20075252 : applied on Thu Jan 22 14:39:21 EST 2015

    (4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) ADF MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20465665 18820382 20645397 (20747356)... OK
Patch 20747356 : applied on Wed Apr 15 10:18:26 EDT 2015

    (4c) AS Common (/oracle/oem/Middleware12cR4/oracle_common) WEBCENTER PORTAL BUNDLE PATCH 11.1.1.7.1 (16761779)... OK
Patch 16761779 : applied on Wed Apr 15 12:18:20 EDT 2015

    (4d) WebLogic Server (/oracle/oem/Middleware12cR4/wlserver_10.3) 10.3.6.0.11 YUIS Patch (20181997)...     OK
CR(s)..................... 20181997 Jar....................... BUG20181997_1036011.jar Destination............... $WLS_INSTALL_DIR$/bugsfixed/20181997-WLS-10.3.6.0.11_PSU_WebServices-ClientSide-Configuration-README.txt

    (4d) *NEW* WebLogic Server (/oracle/oem/Middleware12cR4/wlserver_10.3) SU Patch [GDFA]: WEBLOGIC.STORE.PERSISTENTSTOREEXCEPTION: [STORE:280040] OCCURS EASILEY (16420963)...     OK
CR(s)..................... 16420963 Jar....................... BUG16420963_1036.jar

    (4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2014-4212 OPMN Patch (19345576)... OK
Patch 19345576 : applied on Thu Jan 22 13:02:25 EST 2015

    (4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2013-3836 PLACEHOLDER FOR SECURITY PATCH FOR WEBCACHE 11.1.1.7.0 WITH OCT2013 CPU (17306880)... OK
Patch 17306880 : applied on Wed Apr 15 10:28:40 EDT 2015

    (4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) CVE-2013-0169,CVE-2011-3389 OSS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2013 (17337741)... OK
Patch 17337741 : applied on Wed Apr 15 10:36:26 EDT 2015

    (4e) WebTier (/oracle/oem/Middleware12cR4/Oracle_WT) WLSPLUGINS (OHS) SECURITY PATCH UPDATE 11.1.1.7.0 CPUJUL2014 (18423831)... OK
Patch 18423831 : applied on Wed Apr 15 12:45:02 EDT 2015

    (4f) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) DB PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804122)... OK
20804122;EM DB PLUGIN BUNDLE PATCH 12.1.0.7.3 20822866,20613886,20804213,20804122

    (4g) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) FMW PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804213)... OK
20804213;EM FMW PLUGIN BUNDLE PATCH 12.1.0.7.3 20822866,20613886,20804213,20804122

    (4h) OMS (/oracle/oem/Middleware12cR4/oms) MOS PLUGIN BUNDLE PATCH 12.1.0.6.4 (20613886)... OK
20613886;EM MOS PLUGIN BUNDLE PATCH 12.1.0.6.4 20822866,20613886,20804213,20804122

    (4i) *UPDATED* OMS (/oracle/oem/Middleware12cR4/oms) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.7 (20822866)... OK
20822866;EM EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.7 20822866,20613886,20804213,20804122

    (4j) *UPDATED* OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) EM-AGENT BUNDLE PATCH 12.1.0.4.8 (20822942)... OK
Patch 20822942 : applied on Thu Apr 30 15:44:14 EDT 2015 20822942, 20354975, 18748821, 19806973, 20423395, 18410745, 20218660

    (4j) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) Merge Patch (18502187)... OK
Patch 18502187 : applied on Fri Apr 03 09:45:56 EDT 2015

    (4j) OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) JDBC Security Patch (18721761)... OK
Patch 18721761 : applied on Fri Apr 03 09:45:52 EDT 2015

    (4j) *NEW* OMS CHAINED AGENT HOME (/oracle/oem/agent12c/core/12.1.0.4.0) CVE 2012-3137 EM Agent only: Instant Client Security Patch (20114054)... OK
Patch 20114054 : applied on Fri May 01 10:01:01 EDT 2015 20114054

    (4k) OMS CHAINED AGENT DB PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.7.0) DB PLUGIN BUNDLE 12.1.0.7.3 AGENT-SIDE (20844888)... OK
Patch 20844888 : applied on Wed Apr 15 08:42:51 EDT 2015 20362561, 20487484, 20676926, 20844888, 19813287, 20653069, 20827168

    (4l) *UPDATED* OMS CHAINED AGENT FMW PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0) FMW PLUGIN BUNDLE 12.1.0.7.3 AGENT-SIDE MONITORING (20844953)... OK
Patch 20844953 : applied on Fri May 01 08:58:19 EDT 2015 20487548, 20511329, 20844953, 20644295, 20407536, 20677020, 19855551, 20455785

    (4l) OMS CHAINED AGENT FMW PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.emas.discovery.plugin_12.1.0.7.0) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE DISCOVERY (20677038)... OK
Patch 20677038 : applied on Wed Apr 01 09:51:39 EDT 2015 20644315, 20677038

    (4m) OMS CHAINED AGENT BEACON PLUGIN (/oracle/oem/agent12c/core/12.1.0.4.0/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.4.0) EM-BEACON BUNDLE PATCH 12.1.0.4.1 (20466772)... OK
Patch 20466772 : applied on Wed Apr 15 13:13:29 EDT 2015 20466772, 20397739

    (4n) *NEW* OMS CHAINED AGENT EM-OH BUNDLE PATCH 12.1.0.4.1 (20855134)... OK
Patch 20855134 : applied on Thu Apr 30 15:54:47 EDT 2015 15985793, 20855134

    (4o) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) PSU 11.2.0.4.6 (APR2015) (20299013)... OK
Patch 20299013 : applied on Wed Apr 15 09:30:51 EDT 2015 Patch description: "Database Patch Set Update : 11.2.0.4.6 (20299013)"

    (4o) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/11.2.0/dbhome_2) ORACLE JAVAVM COMPONENT 11.2.0.4.3 DATABASE PSU (APR2015) (20406239)... OK
Patch 20406239 : applied on Wed Apr 15 09:26:41 EDT 2015


(5) Checking EM12c Java versions against baseline (see notes 1506916.1, 1492980.1)

    (5a) *NEW* MW (/oracle/oem/Middleware12cR4/jdk16/jdk) Java version 1.6.0_95 (9553040)...     OK

    (5b) *NEW* WebTier (/oracle/oem/Middleware12cR4/Oracle_WT/jdk) Java version 1.6.0_95 (9553040)...     OK

Failed test count: 2 - Review output
Visit https://pardydba.wordpress.com/2015/03/09/em12c-r4-ssl-security-checkup-script/ for the latest version.

Body of script:

#!/bin/bash
#
# This script should examine your EM12c R4 environment, identify the ports
# each component uses, and check for SSLv2/SSLv3 usage, as well as make
# sure that weak cipher suites get rejected.  It also contains a patch
# check currently comparing against the latest recommended patches
# and flags the use of self-signed certificates.  Further checks include
# EM12c Java JDK version.
#
# Added in v1.0:  Repository database patch check
# Added in v1.1:  EM12c Java JDK version check
# Change in v1.2: Removed patch 19948000 recommendation for OHS.
# Change in v1.3: Update for 30 Apr 2015 patches, add EM-OH plugin home
#                 restored GDFA/16420963 for WLS
#                 added 20114054 for Agent - only applicable for Linux x86-64
#
# From: @BrianPardy on Twitter
#
# Known functional on Linux x86-64, Solaris, AIX.
#
# Run this script as the Oracle EM12c software owner, with your environment
# fully up and running.
#
# Thanks to Dave Corsar, who tested on Solaris and let me know the
# changes needed to make an earlier version work on Solaris.
#
# Thanks to opa tropa who confirmed AIX functionality and noted the
# use of GNU extensions to grep, which I have since removed.
#
# Dedicated to our two Lhasa Apsos, Lucy (RIP 3/13/2015) and Ethel.
#

SCRIPTNAME=`basename $0`
PATCHDATE="30 April 2015"
OMSHOST=`hostname -f`
VERSION="1.3"
FAIL_COUNT=0

RUN_DB_CHECK=0

HOST_OS=`uname -s`
HOST_ARCH=`uname -m`

ORAGCHOMELIST="/etc/oragchomelist"
ORATAB="/etc/oratab"

if [[ ! -r $ORAGCHOMELIST ]]; then            # Solaris
    ORAGCHOMELIST="/var/opt/oracle/oragchomelist"
fi

if [[ ! -r $ORATAB ]]; then                 # Solaris
    ORATAB="/var/opt/oracle/oratab"
fi

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
    GREP=/usr/sfw/bin/gegrep
else
    GREP=`which grep`
fi

OMS_HOME=`$GREP -i oms $ORAGCHOMELIST | xargs ls -d 2>/dev/null`

OPATCH="$OMS_HOME/OPatch/opatch"
OPATCHAUTO="$OMS_HOME/OPatch/opatchauto"
OMSORAINST="$OMS_HOME/oraInst.loc"
ORAINVENTORY=`head -n 1 $OMSORAINST | awk -F= '{print $2}'`

MW_HOME=`dirname $OMS_HOME`
BIP_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"Oracle_BI" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
COMMON_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"common" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
WEBTIER_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"webtier" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
AGENT_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"agent12c" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`
AGENT_DB_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.7.0"
AGENT_FMW_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0"
AGENT_FMW_PLUGIN_DISC_HOME="$AGENT_HOME/../../plugins/oracle.sysman.emas.discovery.plugin_12.1.0.7.0"
AGENT_BEACON_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.beacon.agent.plugin_12.1.0.4.0"
AGENT_OH_PLUGIN_HOME="$AGENT_HOME/../../plugins/oracle.sysman.oh.agent.plugin_12.1.0.4.0"

EM_INSTANCE_BASE=`$GREP GCDomain $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/user_projects.*$//' | sed -e 's/"//'`
WL_HOME=`$GREP wlserver $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/samples.*$//' | sed -e 's/"//' | uniq`

EMGC_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties"
EMBIP_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/embip.properties"
OPMN_PROPS="$EM_INSTANCE_BASE/WebTierIH1/config/OPMN/opmn/ports.prop"
OHS_ADMIN_CONF="$EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/admin.conf"

PORT_UPL=`$GREP EM_UPLOAD_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS=`$GREP EM_CONSOLE_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS_JAVA=`$GREP MS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_NODEMANAGER=`$GREP EM_NODEMGR_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_BIP=`$GREP BIP_HTTPS_PORT $EMBIP_PROPS | awk -F= '{print $2}'`
PORT_ADMINSERVER=`$GREP AS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OPMN=`$GREP '/opmn/remote_port' $OPMN_PROPS | awk -F= '{print $2}'`
PORT_OHS_ADMIN=`$GREP Listen $OHS_ADMIN_CONF | awk '{print $2}'`
PORT_AGENT=`$AGENT_HOME/bin/emctl status agent | $GREP 'Agent URL' | sed -e 's/\/emd\/main\///' | sed -e 's/^.*://' | uniq`

REPOS_DB_CONNDESC=`$GREP EM_REPOS_CONNECTDESCRIPTOR $EMGC_PROPS | sed -e 's/EM_REPOS_CONNECTDESCRIPTOR=//' | sed -e 's/\\\\//g'`
REPOS_DB_HOST=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*HOST=//' | sed -e 's/).*$//'`
REPOS_DB_SID=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*SID=//' | sed -e 's/).*$//'`

if [[ "$REPOS_DB_HOST" == "$OMSHOST" ]]; then
    REPOS_DB_HOME=`$GREP "$REPOS_DB_SID:" $ORATAB | awk -F: '{print $2}'`
    REPOS_DB_VERSION=`$REPOS_DB_HOME/OPatch/opatch lsinventory -oh $REPOS_DB_HOME | $GREP 'Oracle Database' | awk '{print $4}'`

    if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
        RUN_DB_CHECK=1
    fi

    if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
        RUN_DB_CHECK=1
    fi

    if [[ "$RUN_DB_CHECK" -eq 0 ]]; then
        echo -e "\tSkipping local repository DB patch check, only 11.2.0.4 or 12.1.0.2 supported by this script for now"
    fi
fi




sslcheck () {
    OPENSSL_CHECK_COMPONENT=$1
    OPENSSL_CHECK_HOST=$2
    OPENSSL_CHECK_PORT=$3
    OPENSSL_CHECK_PROTO=$4

    OPENSSL_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -$OPENSSL_CHECK_PROTO 2>&1 | $GREP Cipher | $GREP -c 0000`
    
    

    if [[ $OPENSSL_CHECK_PROTO == "tls1" ]]; then
        echo -en "\tConfirming $OPENSSL_CHECK_PROTO available for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
        if [[ $OPENSSL_RETURN -eq "0" ]]; then
            echo OK
        else
            echo FAILED
            FAIL_COUNT=$((FAIL_COUNT+1))
        fi
    fi

    if [[ $OPENSSL_CHECK_PROTO == "ssl2" || $OPENSSL_CHECK_PROTO == "ssl3" ]]; then
        echo -en "\tConfirming $OPENSSL_CHECK_PROTO disabled for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
        if [[ $OPENSSL_RETURN -ne "0" ]]; then
            echo OK
        else
            echo FAILED
            FAIL_COUNT=$((FAIL_COUNT+1))
        fi
    fi
}

opatchcheck () {
    OPATCH_CHECK_COMPONENT=$1
    OPATCH_CHECK_OH=$2
    OPATCH_CHECK_PATCH=$3

    if [[ "$OPATCH_CHECK_COMPONENT" == "ReposDBHome" ]]; then
        OPATCH_RET=`$OPATCH_CHECK_OH/OPatch/opatch lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
    else
        OPATCH_RET=`$OPATCH lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
    fi

    if [[ -z "$OPATCH_RET" ]]; then
        echo FAILED
        FAIL_COUNT=$((FAIL_COUNT+1))
    else
        echo OK
    fi

    echo $OPATCH_RET

}

opatchautocheck () {
    OPATCHAUTO_CHECK_COMPONENT=$1
    OPATCHAUTO_CHECK_OH=$2
    OPATCHAUTO_CHECK_PATCH=$3

    OPATCHAUTO_RET=`$OPATCHAUTO lspatches -oh $OPATCHAUTO_CHECK_OH | $GREP $OPATCHAUTO_CHECK_PATCH`

    if [[ -z "$OPATCHAUTO_RET" ]]; then
        echo FAILED
        FAIL_COUNT=$((FAIL_COUNT+1))
    else
        echo OK
    fi

    echo $OPATCHAUTO_RET

}

certcheck () {
    CERTCHECK_CHECK_COMPONENT=$1
    CERTCHECK_CHECK_HOST=$2
    CERTCHECK_CHECK_PORT=$3

    echo -ne "\tChecking certificate at $CERTCHECK_CHECK_COMPONENT ($CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT)... "

    OPENSSL_SELFSIGNED_COUNT=`echo Q | openssl s_client -prexit -connect $CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT 2>&1 | $GREP -ci "self signed certificate"`

    if [[ $OPENSSL_SELFSIGNED_COUNT -eq "0" ]]; then
        echo OK
    else
        echo FAILED - Found self-signed certificate
        FAIL_COUNT=$((FAIL_COUNT+1))
    fi
}


ciphercheck () {
    OPENSSL_CHECK_COMPONENT=$1
    OPENSSL_CHECK_HOST=$2
    OPENSSL_CHECK_PORT=$3

    echo -ne "\tChecking LOW strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

    OPENSSL_LOW_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher LOW 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

    if [[ $OPENSSL_LOW_RETURN -eq "0" ]]; then
        echo -e "\tFAILED - PERMITS LOW STRENGTH CIPHER CONNECTIONS"
        FAIL_COUNT=$((FAIL_COUNT+1))
    else
        echo -e "\tOK"
    fi


    echo -ne "\tChecking MEDIUM strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

    OPENSSL_MEDIUM_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher MEDIUM 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

    if [[ $OPENSSL_MEDIUM_RETURN -eq "0" ]]; then
        echo -e "\tFAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS"
        FAIL_COUNT=$((FAIL_COUNT+1))
    else
        echo -e "\tOK"
    fi



    echo -ne "\tChecking HIGH strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

    OPENSSL_HIGH_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher HIGH 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

    if [[ $OPENSSL_HIGH_RETURN -eq "0" ]]; then
        echo -e "\tOK"
    else
        echo -e "\tFAILED - CANNOT CONNECT WITH HIGH STRENGTH CIPHER"
        FAIL_COUNT=$((FAIL_COUNT+1))
    fi
    echo
}

wlspatchcheck () {
    WLSDIR=$1
    PATCH=$2

    WLSCHECK_RETURN=`( cd $MW_HOME/utils/bsu && $MW_HOME/utils/bsu/bsu.sh -report ) | $GREP $PATCH`
    WLSCHECK_COUNT=`echo $WLSCHECK_RETURN | wc -l`

    if [[ $WLSCHECK_COUNT -ge "1" ]]; then
        echo -e "\tOK"
    else
        echo -e "\tFAILED - PATCH NOT FOUND"
        FAIL_COUNT=$((FAIL_COUNT+1))
    fi

    echo $WLSCHECK_RETURN
    
}

javacheck () {
    WHICH_JAVA=$1
    JAVA_DIR=$2

    JAVACHECK_RETURN=`$JAVA_DIR/bin/java -version 2>&1 | $GREP version | awk '{print $3}' | sed -e 's/"//g'`

    if [[ "$JAVACHECK_RETURN" == "1.6.0_95" ]]; then
        echo -e "\tOK"
    else
        echo -e "\tFAILED - Found version $JAVACHECK_RETURN"
        FAIL_COUNT=$((FAIL_COUNT+1))
    fi
}


### MAIN SCRIPT HERE


echo -e "Performing EM12cR4 security checkup version $VERSION.\n"

echo "Using port definitions from configuration files "
echo -e "\t/etc/oragchomelist"
echo -e "\t$EMGC_PROPS"
echo -e "\t$EMBIP_PROPS"
echo -e "\t$OPMN_PROPS"
echo -e "\t$OHS_ADMIN_CONF"
echo
echo -e "\tAgent port found at $OMSHOST:$PORT_AGENT"
echo -e "\tBIPublisher port found at $OMSHOST:$PORT_BIP"
echo -e "\tNodeManager port found at $OMSHOST:$PORT_NODEMANAGER"
echo -e "\tOHSadmin port found at $OMSHOST:$PORT_OHS_ADMIN"
echo -e "\tOMSconsole port found at $OMSHOST:$PORT_OMS"
echo -e "\tOMSproxy port found at $OMSHOST:$PORT_OMS_JAVA"
echo -e "\tOMSupload port found at $OMSHOST:$PORT_UPL"
echo -e "\tOPMN port found at $OMSHOST:$PORT_OPMN"
echo -e "\tWLSadmin found at $OMSHOST:$PORT_ADMINSERVER"
echo
echo -e "\tRepository DB version=$REPOS_DB_VERSION SID=$REPOS_DB_SID host=$REPOS_DB_HOST"

if [[ $RUN_DB_CHECK -eq "1" ]]; then
    echo -e "\tRepository DB on OMS server, will check patches in $REPOS_DB_HOME"

fi

#echo -e "\tRepository DB connect descriptor $REPOS_DB_CONNDESC"
#echo -e "\tRepository DB host $REPOS_DB_HOST"
#echo -e "\tRepository DB SID $REPOS_DB_SID"
#echo -e "\tRepository DB ORACLE_HOME $REPOS_DB_HOME"



echo -e "\n(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)"

echo -e "\n\t(1a) Forbid SSLv2 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl2
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl2
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl2
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN ssl2
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl2
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl2
sslcheck OMSupload $OMSHOST $PORT_UPL ssl2
sslcheck OPMN $OMSHOST $PORT_OPMN ssl2
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl2

echo -e "\n\t(1b) Forbid SSLv3 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl3
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl3
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl3
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN ssl3
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl3
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl3
sslcheck OMSupload $OMSHOST $PORT_UPL ssl3
sslcheck OPMN $OMSHOST $PORT_OPMN ssl3
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl3

echo -e "\n\t(1c) Permit TLSv1 connections"
sslcheck Agent $OMSHOST $PORT_AGENT tls1
sslcheck BIPublisher $OMSHOST $PORT_BIP tls1
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER tls1
sslcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN tls1
sslcheck OMSconsole $OMSHOST $PORT_OMS tls1
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA tls1
sslcheck OMSupload $OMSHOST $PORT_UPL tls1
sslcheck OPMN $OMSHOST $PORT_OPMN tls1
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER tls1

echo -e "\n(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)"
ciphercheck Agent $OMSHOST $PORT_AGENT
ciphercheck BIPublisher $OMSHOST $PORT_BIP
ciphercheck NodeManager $OMSHOST $PORT_NODEMANAGER
ciphercheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
ciphercheck OMSconsole $OMSHOST $PORT_OMS
ciphercheck OMSproxy $OMSHOST $PORT_OMS_JAVA
ciphercheck OMSupload $OMSHOST $PORT_UPL
ciphercheck OPMN $OMSHOST $PORT_OPMN
ciphercheck WLSadmin $OMSHOST $PORT_ADMINSERVER

echo -e "\n(3) Checking self-signed certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)"
certcheck Agent $OMSHOST $PORT_AGENT
certcheck BIPublisher $OMSHOST $PORT_BIP
certcheck NodeManager $OMSHOST $PORT_NODEMANAGER
certcheck OHSadmin $OMSHOST $PORT_OHS_ADMIN
certcheck OMSconsole $OMSHOST $PORT_OMS
certcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
certcheck OMSupload $OMSHOST $PORT_UPL
certcheck OPMN $OMSHOST $PORT_OPMN
certcheck WLSadmin $OMSHOST $PORT_ADMINSERVER



echo -e "\n(4) Checking EM12c Oracle home patch levels against $PATCHDATE baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)"

#echo -ne "\n\t(4a) OMS ($OMS_HOME) PSU2 Patch 19830994... "
#opatchcheck OMS $OMS_HOME 19830994

echo -ne "\n\t(4a) OMS ($OMS_HOME) ENTERPRISE MANAGER BASE PLATFORM - OMS 12.1.0.4.3 PSU Patch (20392036)... "
opatchcheck OMS $OMS_HOME 20392036

echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) CPUJAN2015 Patch (19822893)... "
opatchcheck BIP $BIP_HOME 19822893

echo -ne "\n\t(4b) BI Publisher ($BIP_HOME) Merge Patch (20444447)... "
opatchcheck BIP $BIP_HOME 20444447

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) CVE-2015-0426 Oracle Help Patch (20075252)... "
opatchcheck COMMON $COMMON_HOME 20075252

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) ADF MERGE REQUEST ON TOP OF 11.1.1.7.1 FOR BUGS 20465665 18820382 20645397 (20747356)... "
opatchcheck COMMON $COMMON_HOME 20747356

echo -ne "\n\t(4c) AS Common ($COMMON_HOME) WEBCENTER PORTAL BUNDLE PATCH 11.1.1.7.1 (16761779)... "
opatchcheck COMMON $COMMON_HOME 16761779

#echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.10 12UV Patch (19637463)... "
#wlspatchcheck $WL_HOME 19637463

echo -ne "\n\t(4d) WebLogic Server ($WL_HOME) 10.3.6.0.11 YUIS Patch (20181997)... "
wlspatchcheck $WL_HOME 20181997

echo -ne "\n\t(4d) *NEW* WebLogic Server ($WL_HOME) SU Patch [GDFA]: WEBLOGIC.STORE.PERSISTENTSTOREEXCEPTION: [STORE:280040] OCCURS EASILEY (16420963)... "
wlspatchcheck $WL_HOME 16420963

# Commented this patch out 4/17/2015, as Oracle no longer recommends it for EM12c installations.
#echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CPUJAN2015 Patch (19948000)... "
#opatchcheck WebTier $WEBTIER_HOME 19948000

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2014-4212 OPMN Patch (19345576)... "
opatchcheck WebTier $WEBTIER_HOME 19345576

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2013-3836 PLACEHOLDER FOR SECURITY PATCH FOR WEBCACHE 11.1.1.7.0 WITH OCT2013 CPU (17306880)... "
opatchcheck WebTier $WEBTIER_HOME 17306880

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) CVE-2013-0169,CVE-2011-3389 OSS SECURITY PATCH UPDATE 11.1.1.7.0 CPUOCT2013 (17337741)... "
opatchcheck WebTier $WEBTIER_HOME 17337741

echo -ne "\n\t(4e) WebTier ($WEBTIER_HOME) WLSPLUGINS (OHS) SECURITY PATCH UPDATE 11.1.1.7.0 CPUJUL2014 (18423831)... "
opatchcheck WebTier $WEBTIER_HOME 18423831

#echo -ne "\n\t(4f) *UPDATED* OMS ($OMS_HOME) DB PLUGIN BUNDLE 12.1.0.7.2 (20613714)... "
#opatchautocheck OMS $OMS_HOME 20613714

echo -ne "\n\t(4f) *UPDATED* OMS ($OMS_HOME) DB PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804122)... "
opatchautocheck OMS $OMS_HOME 20804122

#echo -ne "\n\t(4g) *UPDATED* OMS ($OMS_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 (20613870)... "
#opatchautocheck OMS $OMS_HOME 20613870

echo -ne "\n\t(4g) *UPDATED* OMS ($OMS_HOME) FMW PLUGIN BUNDLE PATCH 12.1.0.7.3 (20804213)... "
opatchautocheck OMS $OMS_HOME 20804213


echo -ne "\n\t(4h) OMS ($OMS_HOME) MOS PLUGIN BUNDLE PATCH 12.1.0.6.4 (20613886)... "
opatchautocheck OMS $OMS_HOME 20613886

#echo -ne "\n\t(4i) *UPDATED* OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE 12.1.0.6.6 (20613853)... "
#opatchautocheck OMS $OMS_HOME 20613853

echo -ne "\n\t(4i) *UPDATED* OMS ($OMS_HOME) EXADATA PLUGIN BUNDLE PATCH 12.1.0.6.7 (20822866)... "
opatchautocheck OMS $OMS_HOME 20822866

#echo -ne "\n\t(4j) *UPDATED* OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE 12.1.0.4.7 (20613931)... "
#opatchcheck Agent $AGENT_HOME 20613931

echo -ne "\n\t(4j) *UPDATED* OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 12.1.0.4.8 (20822942)... "
opatchcheck Agent $AGENT_HOME 20822942

echo -ne "\n\t(4j) OMS CHAINED AGENT HOME ($AGENT_HOME) Merge Patch (18502187)... "
opatchcheck Agent $AGENT_HOME 18502187

echo -ne "\n\t(4j) OMS CHAINED AGENT HOME ($AGENT_HOME) JDBC Security Patch (18721761)... "
opatchcheck Agent $AGENT_HOME 18721761

if [[ "$HOST_OS" == "Linux" && "$HOST_ARCH" == "x86_64" ]]; then
    echo -ne "\n\t(4j) *NEW* OMS CHAINED AGENT HOME ($AGENT_HOME) CVE 2012-3137 EM Agent only: Instant Client Security Patch (20114054)... "
    opatchcheck Agent $AGENT_HOME 20114054
fi

#echo -ne "\n\t(4k) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE 20676926... "
#opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 20676926

echo -ne "\n\t(4k) OMS CHAINED AGENT DB PLUGIN ($AGENT_DB_PLUGIN_HOME) DB PLUGIN BUNDLE 12.1.0.7.3 AGENT-SIDE (20844888)... "
opatchcheck AgentDBPlugin $AGENT_DB_PLUGIN_HOME 20844888

#echo -ne "\n\t(4l) *UPDATED* OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE MONITORING (20677020)... "
#opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 20677020

echo -ne "\n\t(4l) *UPDATED* OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_HOME) FMW PLUGIN BUNDLE 12.1.0.7.3 AGENT-SIDE MONITORING (20844953)... "
opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_HOME 20844953

echo -ne "\n\t(4l) OMS CHAINED AGENT FMW PLUGIN ($AGENT_FMW_PLUGIN_DISC_HOME) FMW PLUGIN BUNDLE 12.1.0.7.2 AGENT-SIDE DISCOVERY (20677038)... "
opatchcheck AgentFMWPlugin $AGENT_FMW_PLUGIN_DISC_HOME 20677038

echo -ne "\n\t(4m) OMS CHAINED AGENT BEACON PLUGIN ($AGENT_BEACON_PLUGIN_HOME) EM-BEACON BUNDLE PATCH 12.1.0.4.1 (20466772)... "
opatchcheck AgentBeaconPlugin $AGENT_BEACON_PLUGIN_HOME 20466772

echo -ne "\n\t(4n) *NEW* OMS CHAINED AGENT EM-OH BUNDLE PATCH 12.1.0.4.1 (20855134)... "
opatchcheck AgentOHPlugin $AGENT_OH_PLUGIN_HOME 20855134



if [[ $RUN_DB_CHECK -eq 1 ]]; then

#    if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
#        echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.5 19769489... "
#        opatchcheck ReposDBHome $REPOS_DB_HOME 19769489
#
#        echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.2 DATABASE PSU (JAN2015) 19877440... "
#        opatchcheck ReposDBHome $REPOS_DB_HOME 19877440
#    fi

    if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
        echo -ne "\n\t(4o) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 11.2.0.4.6 (APR2015) (20299013)... "
        opatchcheck ReposDBHome $REPOS_DB_HOME 20299013

        echo -ne "\n\t(4o) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 11.2.0.4.3 DATABASE PSU (APR2015) (20406239)... "
        opatchcheck ReposDBHome $REPOS_DB_HOME 20406239
    fi

#    if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
#        echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) Required Patch 20243268... "
#        opatchcheck ReposDBHome $REPOS_DB_HOME 20243268
#
#        echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.2 19769480... "
#        opatchcheck ReposDBHome $REPOS_DB_HOME 19769480
#
#        echo -ne "\n\t(4m) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.2 ORACLE JAVAVM COMPONENT 12.1.0.2.2 DATABASE PSU (JAN2015) 19877336... "
#        opatchcheck ReposDBHome $REPOS_DB_HOME 19877336
#    fi

    if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
        echo -ne "\n\t(4o) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) Required Patch (20243268)... "
        opatchcheck ReposDBHome $REPOS_DB_HOME 20243268

        echo -ne "\n\t(4o) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.3 (APR2015) (20299023)... "
        opatchcheck ReposDBHome $REPOS_DB_HOME 20299023

        echo -ne "\n\t(4o) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.3 DATABASE PSU (APR2015) (20415564)... "
        opatchcheck ReposDBHome $REPOS_DB_HOME 20415564
    fi
fi

echo

echo -e "\n(5) Checking EM12c Java versions against baseline (see notes 1506916.1, 1492980.1)"

echo -ne "\n\t(5a) *NEW* MW ($MW_HOME/jdk16/jdk) Java version 1.6.0_95 (9553040)... "
javacheck MW $MW_HOME/jdk16/jdk 1.6.0_95

echo -ne "\n\t(5b) *NEW* WebTier ($WEBTIER_HOME/jdk) Java version 1.6.0_95 (9553040)... "
javacheck WebTier $WEBTIER_HOME/jdk 1.6.0_95

echo

if [[ $FAIL_COUNT -gt "0" ]]; then
    echo "Failed test count: $FAIL_COUNT - Review output"
else
    echo "All tests succeeded."
fi

echo "Visit https://pardydba.wordpress.com/2015/03/09/em12c-r4-ssl-security-checkup-script/ for the latest version."

echo

exit

If you try this script, please leave me a comment.  If you can share any changes you’ve made that allow it to run on other operating systems, I and others will appreciate it. I spent a lot of time making it so the user does not have to specify any directory locations or port settings, so if you have code changes to offer please let me know.  If enough people use this I may learn how to put it on github or something.

Good luck and happy compliance audits!

Further Reading

• Patching 101 – The User Friendly Guide To Understanding EM Patches
• Step by step: Configuring third party SSL/TLS certificates in EM12c R4

• How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack

This post documents an unusual issue I encountered with the Oracle HTTP Server (OHS) installation in my Oracle Enterprise Manager 12c R4 (12.1.0.4) environment after following MOS note 1984662.1 and applying patch 19948000 (CPUJAN2015) to my OHS home.  It also contains a workaround I found that you should consider UNSUPPORTED, UNOFFICIAL, and NOT RECOMMENDED, only for use if absolutely necessary to meet auditor requirements.  If you do not have to follow the steps I describe below, I suggest waiting for new patches and further guidance from Oracle Support. If this change breaks your system and eats all the food in the break room refrigerator, I warned you not to do it.

Like other security-conscious EM12c admins, I want to keep my installation secure, and so I watch closely when security patches become available for EM12c or its various components. Thus, when I noticed patch 19948000’s availability for OHS, which disables SSLv3, I installed it on my system, and confirmed through testing that OHS no longer permitted SSLv3 connections (test for yourself with: openssl s_client -connect host.domain.com:port -ssl3, or try my EM12c SSL security checkup script that I have blogged about previously).

As I proceeded with further hardening of my EM12c system, specifically an attempt to disable LOW and MEDIUM strength cipher suite usage as per MOS note 1477287.1, I noticed that after following the directions provided, all of my EM12c endpoints correctly rejected LOW and MEDIUM strength ciphers, with one exception.  The OMS HTTPS upload port, inexplicably, continued to permit LOW strength connections. It refused MEDIUM strength ciphers, but had no problem accepting a LOW strength DES-CBC-SHA connection over TLSv1:

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW
[...]
SSL handshake has read 4109 bytes and written 385 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA
    Session-ID: 37BF30668DCAD2CC5D0BAC4142CC1FA1
    Session-ID-ctx:
    Master-Key: [redacted]
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1429290250
    Timeout   : 300 (sec)
---

This confused me greatly, as I had edited all configuration files as instructed, and none of my other OHS listen ports accepted this LOW strength cipher connection.  I spent quite a bit of time trying to diagnose and resolve the issue with no luck, until I eventually stumbled upon an odd fix.  If I remove or comment out the “IfDefine SSL” directives from my $GC_INSTANCE_HOME/WebTierIH1/config/OHS/ohs1/httpd_em.conf file, then suddenly OHS would refuse LOW strength cipher connections on this port, with no apparent ill effect on the other listening ports.

$ openssl s_client -connect omshost.domain.com:4902 -cipher LOW
CONNECTED(00000003)
2282780:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 67 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---

I have noted these IfDefine SSL directives with “HERE” in the excerpt below from my httpd_em.conf file.

##
## CAUTION: Edit only the .template version of this file!
##
##     The command
##         emctl secure [lock|unlock]
##     will replace httpd_em.conf (discarding your changes)
##     using the httpd_em.conf.template file.
##
## This file contains virtual hosts and other directives
## required for the "Enterprise Manager Central Console"
## to function correctly.
##

#UseWebCacheIp On

<IfDefine SSL>      #### HERE
    Listen 4902
    <VirtualHost *:4902>
        <Location /empbs/upload>
            Order allow,deny
            Allow from all
        </Location>
        <Location /empbs/jobrecv>
            Order allow,deny
            Allow from all
        </Location>
        <Location /em>
            Order allow,deny
            Allow from all
        </Location>
        <Location /agent_download>
            Order allow,deny
            Allow from all
        </Location>
        <Location /xmlpserver>
            Order allow,deny
            Allow from all
        </Location>

        #DocumentRoot &ORACLE_HOME&/Apache/Apache/htdocs
        ServerName omshost.domain.com
        #Port 4902
        Timeout 900

        LogFormat "%h %l %u %t \"%r\" %>s %b [ecid: %{ECID-Context}i] [User-Agent: %{User-Agent}i]" common
        SetEnvIf Request_URI "\.(bmp|jpg|png|gif|css|js$)" no-log
        SetEnvIf Request_URI "/em/dynamicImage/*"  no-log
        CustomLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 10M 100M" common env=!no-log

        ErrorLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs /oracle/oem/gc_inst1/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_error_log 10M 100M"
        SSLEngine on
        SSLCipherSuite HIGH
        SSLWallet file:/oracle/oem/gc_inst1/WebTierIH1/config/OHS/ohs1/keystores/upload
        SSLProtocol TLSv1

        <Files ~ "\.(cgi|shtml)$">
            SSLOptions +StdEnvVars
        </Files>
        #<Directory &ORACLE_HOME&/Apache/Apache/cgi-bin>
        #    SSLOptions +StdEnvVars
        #</Directory>
        SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
    </VirtualHost>
</IfDefine>     #### HERE
[remainder of file removed]

If I leave the IfDefine SSL statements in there, my OMS upload port accepts the weak DES-CBC-SHA cipher along with HIGH strength ciphers.  If I remove the IfDefine SSL, my OMS upload port refuses DES-CBC-SHA along with all other LOW/MEDIUM strength ciphers.

This makes no sense, given what I know of OHS and Apache-like products and the way that the handle the SSLCipherSuite configuration directive.

I raised this issue on Twitter and heard back from Andrew Bulloch at Oracle, who graciously spent quite a bit of time attempting to reproduce the issue on his side and working with me to identify the situations in which this behavior occurs.  After much testing, it appears that this behavior only occurs in the following situation:

1. The OHS installed with EM12c R4 has patch 19948000 installed, AND
2. The administrator has installed a third party SSL certificate, replacing the demo certificate used by default, AND
3. The OHS httpd_em.conf contains the “IfDefine SSL” directive.

If I remove my custom certificate, returning the OMS to the demo certificate, the issue disappears, then returns if I reinstall the custom certificate.

If I remove patch 19948000, the issue disappears, and does not return whether I use a custom certificate or a demo certificate.

If I remove the IfDefine SSL directive, the issue disappears, and does not return whether I use a custom certificate, a demo certificate, or whether or not I have patch 19948000 installed.

I attempted to replicate this behavior with an SSL certificate that did not come from a true certificate authority, by using OpenSSL to create a CA, create a cert, sign it, then install it into OHS per the documentation in MOS note 1399293.1, but I could not reproduce it, possibly due to the fact that I used a certificate signed directly by a root CA (as with the demo certificate) instead of a certificate signed by an intermediate chain certificate signed by a root CA, as with the paid-for commercial certificate that revealed the issue. I have not had a chance to test that configuration.

Unfortunately, removing patch 19948000 means that OHS cannot refuse SSLv3 connections, and removing the custom certificate reverts the system back to the demo certificate that I do not wish to use, both of which will represent audit findings in regulated sites.

Due to this issue, I have edited my EM12c security checkup script to remove my recommendation to install patch 19948000, although I still have it installed.  For security reasons, I will leave my system in the workaround state I have described here, as I want SSLv3 disabled, and I want LOW strength cipher suites disabled, and I want to use a custom SSL certificate, but I accept the risk that I may have to undo this setup at any time to receive support or to successfully apply later patches.  You will have to make your own decisions based on your site’s audit requirements and the availability of personnel to validate your configuration and handle future patching.

I would be very interested if anyone else reading this has encountered this issue, as I do not know if my installation somehow uniquely surfaces this behavior or if the certificate vendor that we used has some strange settings on their certificates that cause confusion for OHS.

This post serves to document an issue I encountered after replacing expired SSL/TLS certificates on the server I use for Oracle Enterprise Manager 12c. To put it simply, using opatchauto to apply EM12c PSUs does not work if your WebLogic adminserver has a certificate installed that uses the SHA256 hashing algorithm.

[UPDATED 20151012: Please see this comment and this comment below, from Adam Robinson, who has provided a workaround that may work for you involving editing the opatchauto script to enable JSSE. As always, please consider workarounds requiring you to edit files as unsupported and at your own risk, but I would consider this fix superior to reverting back to the demo certificate every time you need to patch. You will need to repeat this fix every time you update OPatch in your OMS_HOME, though. Adam’s workaround does succeed in my environment.]

Error message

Expect to see the following error when running “opatchauto apply -analyze” or “opatchauto apply” against an installation with an SHA256-hashed certificate on the WLS adminserver:

oracle@omshost:/oracle/stage/21603255> opatchauto apply -analyze -property_file ~/property_file
OPatch Automation Tool
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


OPatchauto version : 11.1.0.12.3
OUI version        : 11.1.0.12.0
Running from       : /oracle/oem/Middleware12cR4/oms
Log file location  : /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatch/opatch2015-09-11_10-57-19AM_1.log

OPatchauto log file: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log



OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.


[ Error during Get weblogic Admin Server information Phase]. Detail: OPatchauto was not able to find right interview inputs.
OPatchauto failed:
OPatchauto failed to establish JMX connection to weblogic server. This could be because of one (or) more of the following reasons:
1. Weblogic admin server URL that manages OMS application may not be right.
2. Weblogic admin server credentials (username, password) may not be right.
3. Virtual host configuration. If OMS, weblogic server are on virtual host configuration, Please make sure to add OPatchAuto.OMS_DISABLE_HOST_CHECK=true to command line and run again. (example: /oracle/oem/Middleware12cR4/oms/OPatch/opatchauto apply -analyze -property_file /home/oracle/property_file -invPtrLoc /oracle/oem/Middleware12cR4/oms/oraInst.loc  OPatchAuto.OMS_DISABLE_HOST_CHECK=true)

Please check above conditions and if error(s) still persist, Please contact Oracle support.

Log file location: /oracle/oem/Middleware12cR4/oms/cfgtoollogs/opatchauto/21603255/opatch_oms_2015-09-11_10-57-22AM_analyze.log

Recommended actions: Please correct the interview inputs and run opatchauto again.

OPatchauto failed with error code 231

Confirmation of the issue

To confirm this issue in your environment after receiving the preceding error message, check the hashing algorithm used on your adminserver certificate. I prefer to use the openssl commandline tool for this. If you don’t know the port used for your adminserver, you can retrieve it from the $EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties file under AS_HTTPS_PORT. If your certificate does not show the usage of SHA256 (or another hash algorithm from the SHA-2 family) as in my example below, you may have a different problem.

oracle@omshost:~> openssl s_client -prexit -connect omshost.domain.com:7103 /dev/null | openssl x509 -text -in /dev/stdin | grep "Signature Algorithm" 2> /dev/null
        Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

Workaround

To work around this issue, you need to (temporarily!) replace the certificate on your WLS adminserver. Now, whenever I need to apply a PSU release, I resecure WLS using the default demonstration certificate, apply the PSU, then replace my original certificate.

oracle@omshost:/oracle/stage/21603255> emctl secure wls -use_demo_cert
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

[install the PSU according to the README instructions, including any post-installation steps]

oracle@omshost:/oracle/stage/21603255> emctl secure wls -wallet /oracle/oem/oemwallet
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Securing WLS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Securing WLS... Successful
Restart OMS using 'emctl stop oms -all' and 'emctl start oms'
If there are multiple OMSs in this environment, perform this configuration on all of them.
oracle@omshost:/oracle/stage/21603255> emctl stop oms -all ; sleep 5 ; emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up

I have not noticed any other EM12c issues using SHA256-hashed certificates. With this workaround, you can both continue to use quality certificates and keep your OMS patched.

I have recently switched to using Oracle Enterprise Manager 13c (EM13c – 13.1), replacing my previous EM12c installation.  I elected to install a clean new environment instead of an upgrade, because my old install had been upgraded repeatedly going back to the initial release of EM12c and I wanted a fresh start.

I encountered only one difficult issue during the process. When I attempted to add one production Oracle XE 11gR2 database target, EM13c could not compute the target’s dynamic properties, leaving the target broken. Since you cannot submit jobs against a broken target, this prevented me from using EM13c to back up this database.  I had no comparable issues with XE as a target under EM12c.

The key metric errors that showed during this process included:
“Metric evaluation error start – Target {oracle_database.SID.domain.com} is broken: Dynamic Category property error,Get dynamic property error,No such metadata – No valid queryDescriptor or executionDescriptor found for target [oracle_database.SID.domain.com$30]”

and for the database system target:

“Metric evaluation error start – Received an exception when evaluating sev_eval_proc for:Target name = SID.domain.com_sys, metric_name = Response, metric_column = Status; Error msg = Target encountered metric erros; at least one member in in metric error”

I enabled debugging for the agent logs and attempted again to add the XE target.  Errors showing up in the logs included:

2016-01-15 12:10:05,905 [1806:4CE3192] DEBUG – Computing of dynamic property: [ComputeVC] is done (1 msec, error=true)

2016-01-15 12:10:06,452 [1806:F917F5F8] DEBUG – Computing of dynamic property: [GetDumpDestination] is done (0 msec, error=true)

2016-01-15 12:10:06,508 [1813:6EEEAC87] DEBUG – Computing of dynamic property: [DeduceAlertLogFile] is done (1 msec, error=true)

2016-01-15 12:11:18,779 [1830:CD3A325D] DEBUG – Error was added to oracle_database.SID.domain.com$23(0|MONITORED|false|true|<UF>): Invalid Input

2016-01-15 12:11:18,779 [1831:3657AE55] DEBUG – abandoning long op “CDProps:oracle_database.SID.domain.com:ComputeVC:GENERIC_TASK:Fri Jan 15 12:11:18 EST 2016”

2016-01-15 12:11:18,780 [1830:CD3A325D] DEBUG – Error during dynamic property ComputeVC calculation: critical=true, missingCatProps=[VersionCategory], missingProps=[VersionCategory] oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Invalid Input

2016-01-15 12:35:38,038 INFO – Finished dynamic properties computation (total time 817 ms). Number of DP computed: 19. Number of DP with error: 3. Number of dynamic properties that were added: 132.

After reviewing the logs carefully (and posting this as a question in the MOS Oracle XE forum – https://community.oracle.com/thread/3892946) I eventually narrowed the issue down to a query that EM13c runs against DBA_REGISTRY_HISTORY in a target database when added. For database versions greater than 11.2 but less than 12.1.0.2, EM13c assumes that DBA_REGISTRY_HISTORY contains a BUNDLE_SERIES column.  This column does not exist in Oracle XE 11gR2, which reports a version string of 11.2.0.2.

This bug should eventually get a fix as EM13c gets patched, but in the meantime if you need to monitor an Oracle XE target with EM13c, the following workaround took care of the problem for me: create a new DBA_REGISTRY_HISTORY table containing a BUNDLE_SERIES column in your monitoring user’s schema in XE.  So, as user DBSNMP on XE, I ran:

SQL> create table dba_registry_history (ACTION_TIME TIMESTAMP(6), ACTION VARCHAR2(30), NAMESPACE VARCHAR2(30), VERSION VARCHAR2(30), ID NUMBER, BUNDLE_SERIES VARCHAR2(30), COMMENTS VARCHAR2(255));

Since one cannot patch XE, the real DBA_REGISTRY_HISTORY view has no rows and so you do not need to populate any data into this new table.

After adding the table, force a recalculation of dynamic properties by running the following against the EM13c management agent on the XE server:

$ emctl reload agent dynamicproperties SID.domain.com:oracle_database
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD recompute dynprops completed successfully

Once that completed successfully my XE target started to show the correct status in EM13c and I can submit jobs against the target.  All fixed.  I recommend deleting the DBSNMP.DBA_REGISTRY_HISTORY table once the bug gets fixed in OEM.

A Friday afternoon on April Fool’s Day seems like a good time to return to my earlier plan to write about liquor as the Data Driven Drinker. So I give you the elderflower vanilla fizz, or the Madonna. This cocktail uses local Vermont ingredients from St Johnsbury and Warren.

The elderflower rum gives a light semisweet flavor up front that compares favorably to a liqueur (e.g. St Germain) . The vanilla rum balances it out and shows on the finish. Vary the amount of ice and club soda according to taste.

Ingredients:

1 ounce Dunc’s Mill Elderflower Rum
1 ounce Mad River Distillers’ Vanilla Rum
1/2 ounce lemon juice
1/2 ounce simple syrup
4 ounces club soda
Ice

Shake rum, lemon juice and simple syrup with ice and strain into a long glass over ice. Top with club soda and garnish with lemon slice.

Elderflower Vanilla Fizz

Introduction

This post continues my series on securing Oracle Enterprise Manager environments with some updates relevant to EM13c. Oracle has made significant security improvements with Oracle Enterprise Manager 13c over the prior 12c version, first released in October 2011, more than four and a half years ago at this point. In the interest of security, I have to strongly recommend that any sites still using EM12c upgrade to (or perform a fresh installation of) EM13c as soon as possible. More recent versions of EM12c like 12.1.0.5 (June 2015) continue to use the same technology stack as the initial release, and the world of security has massively changed since then. Notably, EM13c uses Java 7, WebLogic 12.1.3, and disables SSLv3 out of the box.

Just to recap, back at the EM12c original release date:

• Practically nobody had ever heard of Edward Snowden
• The first release of Java 7 celebrated its three month birthday
• Two months later, Oracle released WebLogic 12c; EM12c users remained on WebLogic 10.3.6
• One month earlier, the public learned of the BEAST attack and people still believed that using RC4 (immune to BEAST) as a workaround improved security (spoiler warning: it did not)
• We had three years to wait before the POODLE vulnerability caused vendors to recognize the need to disable SSLv3 (you DID disable SSLv3, right?)
• Oracle still considered the MD5 hashing algorithm good enough to use in self-signed certificates produced by EM12c, despite flaws known to exist since 1996
• Web browsers considered the SHA-1 hashing algorithm, now also deprecated due to brokenness, good enough to use

As the security world’s known unknowns collapsed around us, proactive EM12c administrators sought to make the best of their lot. Outside of Oracle, I and others poked at the software and wrote blog articles, while inside Oracle effort proceeded to support more recent Java releases that brought with them better cipher suites and hashing algorithms, as well as the usual security fixes. This process took some time for all involved and hit a few bumps along the way.

• Replacement of EM12c certificates with custom certificates making use of the SHA-2 hashing algorithm appeared to work fine until we tried to apply patches: EM12c opatchauto, SHA256, and you
• Each time a new attack broke in the press, proactive EM12c administrators including myself looked for ways to mitigate it, as with my first fumbling efforts when the POODLE bug became public: How to unofficially disable SSL v3 in Oracle Enterprise Manager 12c to mitigate POODLE attack
• The difficulty of keeping track of now-monthly patching and rechecking known issues after patching led me to create an automated security check script for EM12c: EM12c R4 SSL Security Checkup Script
• The official documentation, though exhaustive, sometimes complicated security issues beyond the minimum necessary to achieve a task, so I produced a step-by-step guide to replacing as many certificates as possible with those sourced from third parties or internal sources: Step by step: Configuring third party SSL/TLS certificates in EM12c R4
• Sometimes bizarre bugs led to an inability to disable known weak cipher suites, requiring deep dives into patching and configuration: EM12c OHS, LOW strength ciphers, custom certificates, and patch 1994800 weirdness
• Sometimes we shoot ourselves in the foot by carelessly upgrading components: When proactive EM12c JDK upgrades bite back
• Applying your strong certificates to the full EM12c stack does not eliminate the severe security issues created by Oracle’s decision to include a widely known demo certificate authority with WebLogic: Insecure certificate warning still exists in EM12c WLS logs with third-party certificates applied

I do not intend in this post to review general day-to-day EM13c security design such as user roles or privileges, object level security within OEM, or integration with identity providers like LDAP; only the infrastructure level issues that tend to change in brief large bursts as new attacks come out. See this excellent list of EM13c blogs, links and videos that Philip Brown has provided for more details on these and other items.

On to EM13c

EM13c admins need to keep an eye on the same sorts of items as with EM12c. We really should read the documentation, even if only the Security Guide. I admit I often do not. It contains good information.

Patches

I consider it critical for admins to keep up with the OEM periodic patches, particularly security patches. The script below covers patches up to and including March 31, 2016. I plan to update again after the next set of Oracle security patches arrives, likely mid-April.

Certificates

The process for applying certificates on EM13c does not appear to have changed significantly from the prior version. I have confirmed that the new “omspatcher” tool that replaces opatchauto when applying a system patch to the OMS works perfectly fine with certificates on WebLogic that use the SHA-256 hashing algorithm. Given the upcoming deprecation of SHA-1 across all major browsers I do not see any valid reason not to use SHA-256 with new certificates.

Ciphersuites

Out of the box, my EM13c installation rejected weak ciphersuites and accepted the strong ones. Unfortunately it still accepted some that these versions of Java and OpenSSL consider as MEDIUM strength, so I want to disable those across the entire environment, leaving only the strongest ciphersuites available in this release and permitting other ciphersuites only where necessary.

We will have to live with these unwanted ciphersuites enabled until Oracle provides a supported procedure to disable them across the entire stack. I have performed some preliminary tests and I find it very easy to get OEM into a situation where it cannot startup after manually editing config files that define enabled ciphersuites. The script below will identify ports permitting ciphersuites you may wish to disable when a supported method becomes available.

Security Checkup

Below I provide an early version of the script I use to validate EM13c security configuration. I based this on my earlier EM12c script, linked above. The script will become more useful once I implement patch level checking after release of the first set of EM13c patches, but for the moment it will inspect your EM13c environment to identify relevant ports and confirm that your system will not respond to SSLv2 or SSLv3 requests, does respond to TLSv1 requests, supports HIGH, but not LOW or MEDIUM strength ciphersuites (as defined by the version of OpenSSL installed on your OMS host), and finally checks for the presence of demonstration-not-for-production-use certificates and self-signed certificates.

(A caveat on self-signed certificate checking: OpenSSL, not this script, performs the check, therefore if OpenSSL cannot validate your certificate to a trusted root, this script cannot either. If a well known certification authority has signed your certificates, OpenSSL should validate them successfully, but it may not do so if you use an internal certificate authority to sign certificates. In that case you should install a copy of your internal CA as a trusted root certificate in the system trust store so that OpenSSL can validate your EM13c certificates. I cannot document this process for every OS but Linux users should look to the documentation for the update-ca-certificates or update-ca-trust commands. If my script below incorrectly reports your certificate as self-signed, you can ignore the finding or address the issue at the OpenSSL level.)

EM13c TLS Security Checkup Script

[LATEST UPDATE: 20160406 covering 20160331 EM13c patch bundles]
Consider this version 0.2 beta for the moment. Only tested on Linux x86-64, but may work on AIX and Solaris as the EM12c version I built this upon did work there. Planned future enhancements include checking that you have disabled non-encrypted HTTP access to EM13c components, upgraded Java to the most recent EM13c-supported release, and more.

You can download the initial version of the script from pastebin.


#!/bin/bash
#
# This script should examine your EM13c environment, identify the ports
# each component uses, and check for SSLv2/SSLv3 usage, as well as make
# sure that weak cipher suites get rejected. It will soon contain a patch
# check currently comparing against the latest recommended patches
# and also flags the use of demo or self-signed certificates. Further
# enhancements will include checks for the EM13c Java JDK version.
#
# Released v0.1: Initial beta release 5 Apr 2016
# Changes v0.2: Updated for current patches
#
# From: @BrianPardy on Twitter
#
# Known functional on Linux x86-64, may work on Solaris and AIX.
#
# Run this script as the Oracle EM13c software owner, with your environment
# fully up and running.
#
# Thanks to Dave Corsar, who tested a previous version on Solaris and
# let me know the changes needed to make the script work on Solaris.
#
# Thanks to opa tropa who confirmed AIX functionality on a previous
# version and noted the use of GNU extensions to grep, which I have
# since removed.
#
# Dedicated to our two Lhasa Apsos:
# Lucy (6/13/1998 - 3/13/2015)
# Ethel (6/13/1998 - 7/31/2015)
#
#

SCRIPTNAME=`basename $0`
PATCHDATE="5 Apr 2016"
OMSHOST=`hostname -f`
VERSION="0.1"
FAIL_COUNT=0
FAIL_TESTS=""

RUN_DB_CHECK=0
VERBOSE_CHECKSEC=2

HOST_OS=`uname -s`
HOST_ARCH=`uname -m`

ORAGCHOMELIST="/etc/oragchomelist"
ORATAB="/etc/oratab"

if [[ ! -r $ORAGCHOMELIST ]]; then # Solaris
ORAGCHOMELIST="/var/opt/oracle/oragchomelist"
fi

if [[ ! -r $ORATAB ]]; then # Solaris
ORATAB="/var/opt/oracle/oratab"
fi

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
GREP=/usr/sfw/bin/gegrep
else
GREP=`which grep`
fi

OMS_HOME=`$GREP -i oms $ORAGCHOMELIST | xargs ls -d 2>/dev/null`

if [[ "$OMS_HOME" == "." ]]; then
OMS_HOME=`cat $ORAGCHOMELIST | head -n 1`
fi

OPATCH="$OMS_HOME/OPatch/opatch"
OPATCHAUTO="$OMS_HOME/OPatch/opatchauto"
OMSPATCHER="$OMS_HOME/OMSPatcher/omspatcher"
OMSORAINST="$OMS_HOME/oraInst.loc"
ORAINVENTORY=`$GREP inventory_loc $OMSORAINST | awk -F= '{print $2}'`

MW_HOME=$OMS_HOME
COMMON_HOME="$MW_HOME/oracle_common"

AGENT_HOME=`$GREP -vi REMOVED $ORAINVENTORY/ContentsXML/inventory.xml | $GREP "HOME NAME=\"agent13c" | awk '{print $3}' | sed -e 's/LOC=\"//' | sed -e 's/"//'`

EM_INSTANCE_BASE=`$GREP GCDomain $MW_HOME/domain-registry.xml | sed -e 's/.*=//' | sed -e 's/\/user_projects.*$//' | sed -e 's/"//'`

EMGC_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/emgc.properties"
EMBIP_PROPS="$EM_INSTANCE_BASE/em/EMGC_OMS1/embip.properties"
#OPMN_PROPS="$EM_INSTANCE_BASE/WebTierIH1/config/OPMN/opmn/ports.prop"
#OHS_ADMIN_CONF="$EM_INSTANCE_BASE/WebTierIH1/config/OHS/ohs1/admin.conf"

PORT_UPL=`$GREP EM_UPLOAD_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS=`$GREP EM_CONSOLE_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_OMS_JAVA=`$GREP MS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_NODEMANAGER=`$GREP EM_NODEMGR_PORT $EMGC_PROPS | awk -F= '{print $2}'`
PORT_BIP=`$GREP BIP_HTTPS_PORT $EMBIP_PROPS | awk -F= '{print $2}'`
PORT_BIP_OHS=`$GREP BIP_HTTPS_OHS_PORT $EMBIP_PROPS | awk -F= '{print $2}'`
PORT_ADMINSERVER=`$GREP AS_HTTPS_PORT $EMGC_PROPS | awk -F= '{print $2}'`
#PORT_OPMN=`$GREP '/opmn/remote_port' $OPMN_PROPS | awk -F= '{print $2}'`
#PORT_OHS_ADMIN=`$GREP Listen $OHS_ADMIN_CONF | awk '{print $2}'`
PORT_AGENT=`$AGENT_HOME/bin/emctl status agent | $GREP 'Agent URL' | sed -e 's/\/emd\/main\///' | sed -e 's/^.*://' | uniq`

REPOS_DB_CONNDESC=`$GREP EM_REPOS_CONNECTDESCRIPTOR $EMGC_PROPS | sed -e 's/EM_REPOS_CONNECTDESCRIPTOR=//' | sed -e 's/\\\\//g'`
REPOS_DB_HOST=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*HOST=//' | sed -e 's/).*$//'`
REPOS_DB_SID=`echo $REPOS_DB_CONNDESC | sed -e 's/^.*SID=//' | sed -e 's/).*$//'`

if [[ "$REPOS_DB_HOST" == "$OMSHOST" ]]; then
REPOS_DB_HOME=`$GREP "$REPOS_DB_SID:" $ORATAB | awk -F: '{print $2}'`
REPOS_DB_VERSION=`$REPOS_DB_HOME/OPatch/opatch lsinventory -oh $REPOS_DB_HOME | $GREP 'Oracle Database' | awk '{print $4}'`

if [[ "$REPOS_DB_VERSION" == "11.2.0.4.0" ]]; then
RUN_DB_CHECK=1
fi

if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
RUN_DB_CHECK=1
fi

if [[ "$RUN_DB_CHECK" -eq 0 ]]; then
echo -e "\tSkipping local repository DB patch check, only 11.2.0.4 or 12.1.0.2 supported by this script for now"
fi
fi

sslcheck () {
OPENSSL_CHECK_COMPONENT=$1
OPENSSL_CHECK_HOST=$2
OPENSSL_CHECK_PORT=$3
OPENSSL_CHECK_PROTO=$4

OPENSSL_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -$OPENSSL_CHECK_PROTO 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_CHECK_PROTO == "tls1" ]]; then
echo -en "\tConfirming $OPENSSL_CHECK_PROTO available for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
if [[ $OPENSSL_RETURN -eq "0" ]]; then
echo OK
else
echo FAILED
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:$OPENSSL_CHECK_PROTO protocol connection failed"
fi
fi

if [[ $OPENSSL_CHECK_PROTO == "ssl2" || $OPENSSL_CHECK_PROTO == "ssl3" ]]; then
echo -en "\tConfirming $OPENSSL_CHECK_PROTO disabled for $OPENSSL_CHECK_COMPONENT at $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT... "
if [[ $OPENSSL_RETURN -ne "0" ]]; then
echo OK
else
echo FAILED
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:$OPENSSL_CHECK_PROTO protocol connection succeeded"
fi
fi
}

opatchcheck () {
OPATCH_CHECK_COMPONENT=$1
OPATCH_CHECK_OH=$2
OPATCH_CHECK_PATCH=$3

if [[ "$OPATCH_CHECK_COMPONENT" == "ReposDBHome" ]]; then
OPATCH_RET=`$OPATCH_CHECK_OH/OPatch/opatch lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
else
OPATCH_RET=`$OPATCH lsinv -oh $OPATCH_CHECK_OH | $GREP $OPATCH_CHECK_PATCH`
fi

if [[ -z "$OPATCH_RET" ]]; then
echo FAILED
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPATCH_CHECK_COMPONENT @ ${OPATCH_CHECK_OH}:Patch $OPATCH_CHECK_PATCH not found"
else
echo OK
fi

test $VERBOSE_CHECKSEC -ge 2 && echo $OPATCH_RET

}

opatchautocheck () {
OPATCHAUTO_CHECK_COMPONENT=$1
OPATCHAUTO_CHECK_OH=$2
OPATCHAUTO_CHECK_PATCH=$3

OPATCHAUTO_RET=`$OPATCHAUTO lspatches -oh $OPATCHAUTO_CHECK_OH | $GREP $OPATCHAUTO_CHECK_PATCH`

if [[ -z "$OPATCHAUTO_RET" ]]; then
echo FAILED
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPATCHAUTO_CHECK_COMPONENT @ ${OPATCHAUTO_CHECK_OH}:Patch $OPATCHAUTO_CHECK_PATCH not found"
else
echo OK
fi

test $VERBOSE_CHECKSEC -ge 2 && echo $OPATCHAUTO_RET

}

omspatchercheck () {
OMSPATCHER_CHECK_COMPONENT=$1
OMSPATCHER_CHECK_OH=$2
OMSPATCHER_CHECK_PATCH=$3

OMSPATCHER_RET=`$OMSPATCHER lspatches -oh $OMSPATCHER_CHECK_OH | $GREP $OMSPATCHER_CHECK_PATCH`

if [[ -z "$OMSPATCHER_RET" ]]; then
echo FAILED
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OMSPATCHER_CHECK_COMPONENT @ ${OMSPATCHER_CHECK_OH}:Patch $OMSPATCHER_CHECK_PATCH not found"
else
echo OK
fi

test $VERBOSE_CHECKSEC -ge 2 && echo $OMSPATCHER_RET

}

certcheck () {
CERTCHECK_CHECK_COMPONENT=$1
CERTCHECK_CHECK_HOST=$2
CERTCHECK_CHECK_PORT=$3

echo -ne "\tChecking certificate at $CERTCHECK_CHECK_COMPONENT ($CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT)... "

OPENSSL_SELFSIGNED_COUNT=`echo Q | openssl s_client -prexit -connect $CERTCHECK_CHECK_HOST:$CERTCHECK_CHECK_PORT -tls1 2>&1 | $GREP -ci "self signed certificate"`

if [[ $OPENSSL_SELFSIGNED_COUNT -eq "0" ]]; then
echo OK
else
echo FAILED - Found self-signed certificate
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$CERTCHECK_CHECK_COMPONENT @ ${CERTCHECK_CHECK_HOST}:${CERTCHECK_CHECK_PORT} found self-signed certificate"
fi
}

democertcheck () {
DEMOCERTCHECK_CHECK_COMPONENT=$1
DEMOCERTCHECK_CHECK_HOST=$2
DEMOCERTCHECK_CHECK_PORT=$3

echo -ne "\tChecking demo certificate at $DEMOCERTCHECK_CHECK_COMPONENT ($DEMOCERTCHECK_CHECK_HOST:$DEMOCERTCHECK_CHECK_PORT)... "

OPENSSL_DEMO_COUNT=`echo Q | openssl s_client -prexit -connect $DEMOCERTCHECK_CHECK_HOST:$DEMOCERTCHECK_CHECK_PORT -tls1 2>&1 | $GREP -ci "issuer=/C=US/ST=MyState/L=MyTown/O=MyOrganization/OU=FOR TESTING ONLY/CN"`

if [[ $OPENSSL_DEMO_COUNT -eq "0" ]]; then
echo OK
else
echo FAILED - Found demonstration certificate
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$DEMOCERTCHECK_CHECK_COMPONENT @ ${DEMOCERTCHECK_CHECK_HOST}:${DEMOCERTCHECK_CHECK_PORT} found demonstration certificate"
fi
}

ciphercheck () {
OPENSSL_CHECK_COMPONENT=$1
OPENSSL_CHECK_HOST=$2
OPENSSL_CHECK_PORT=$3

echo -ne "\tChecking LOW strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

OPENSSL_LOW_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher LOW 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

if [[ $OPENSSL_LOW_RETURN -eq "0" ]]; then
echo -e "\tFAILED - PERMITS LOW STRENGTH CIPHER CONNECTIONS"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Permits LOW strength ciphers"
else
echo -e "\tOK"
fi

echo -ne "\tChecking MEDIUM strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

OPENSSL_MEDIUM_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher MEDIUM 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

if [[ $OPENSSL_MEDIUM_RETURN -eq "0" ]]; then
echo -e "\tFAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Permits MEDIUM strength ciphers"
else
echo -e "\tOK"
fi

echo -ne "\tChecking HIGH strength ciphers on $OPENSSL_CHECK_COMPONENT ($OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT)..."

OPENSSL_HIGH_RETURN=`echo Q | openssl s_client -prexit -connect $OPENSSL_CHECK_HOST:$OPENSSL_CHECK_PORT -tls1 -cipher HIGH 2>&1 | $GREP Cipher | uniq | $GREP -c 0000`

if [[ $OPENSSL_HIGH_RETURN -eq "0" ]]; then
echo -e "\tOK"
else
echo -e "\tFAILED - CANNOT CONNECT WITH HIGH STRENGTH CIPHER"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$OPENSSL_CHECK_COMPONENT @ $OPENSSL_CHECK_HOST:${OPENSSL_CHECK_PORT}:Rejects HIGH strength ciphers"
fi
echo
}

wlspatchcheck () {
WLSDIR=$1
WLSPATCH=$2

WLSCHECK_RETURN=`( cd $MW_HOME/utils/bsu && $MW_HOME/utils/bsu/bsu.sh -report ) | $GREP $WLSPATCH`
WLSCHECK_COUNT=`echo $WLSCHECK_RETURN | wc -l`

if [[ $WLSCHECK_COUNT -ge "1" ]]; then
echo -e "\tOK"
else
echo -e "\tFAILED - PATCH NOT FOUND"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WLSDIR:Patch $WLSPATCH not found"
fi

test $VERBOSE_CHECKSEC -ge 2 && echo $WLSCHECK_RETURN

}

javacheck () {
WHICH_JAVA=$1
JAVA_DIR=$2

JAVACHECK_RETURN=`$JAVA_DIR/bin/java -version 2>&1 | $GREP version | awk '{print $3}' | sed -e 's/"//g'`

if [[ "$JAVACHECK_RETURN" == "1.6.0_95" ]]; then
echo -e "\tOK"
else
#echo -e "\tFAILED - Found version $JAVACHECK_RETURN"
echo -e "\tFAILED"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_JAVA Java in ${JAVA_DIR}:Found incorrect version $JAVACHECK_RETURN"
fi
test $VERBOSE_CHECKSEC -ge 2 && echo $JAVACHECK_RETURN
}

paramcheck () {
WHICH_PARAM=$1
WHICH_ORACLE_HOME=$2
WHICH_FILE=$3

PARAMCHECK_RETURN=`$GREP $WHICH_PARAM $WHICH_ORACLE_HOME/network/admin/$WHICH_FILE | awk -F= '{print $2}' | sed -e 's/\s//g'`
if [[ "$WHICH_PARAM" == "SSL_VERSION" ]]; then
if [[ "$PARAMCHECK_RETURN" == "1.0" ]]; then
echo -e "OK"
else
echo -e "FAILED - Found $WHICH_PARAM = $PARAMCHECK_RETURN"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_PARAM in $WHICH_FILE for home ${WHICH_ORACLE_HOME}:incorrect parameter value"
fi
test $VERBOSE_CHECKSEC -ge 2 && echo $PARAMCHECK_RETURN
fi

if [[ "$WHICH_PARAM" == "SSL_CIPHER_SUITES" ]]; then
if [[ "$PARAMCHECK_RETURN" == "(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)" ]]; then
echo -e "OK"
else
echo -e "FAILED - Found $WHICH_PARAM = $PARAMCHECK_RETURN"
FAIL_COUNT=$((FAIL_COUNT+1))
FAIL_TESTS="${FAIL_TESTS}\\n$FUNCNAME:$WHICH_PARAM in $WHICH_FILE for home ${WHICH_ORACLE_HOME}:incorrect parameter value"
fi
test $VERBOSE_CHECKSEC -ge 2 && echo $PARAMCHECK_RETURN
fi
}

### MAIN SCRIPT HERE

echo -e "Performing EM13c security checkup version $VERSION on $OMSHOST at `date`.\n"

echo "Using port definitions from configuration files "
echo -e "\t/etc/oragchomelist"
echo -e "\t$EMGC_PROPS"
echo -e "\t$EMBIP_PROPS"
echo
echo -e "\tAgent port found at $OMSHOST:$PORT_AGENT"
echo -e "\tBIPublisher port found at $OMSHOST:$PORT_BIP"
echo -e "\tBIPublisherOHS port found at $OMSHOST:$PORT_BIP_OHS"
echo -e "\tNodeManager port found at $OMSHOST:$PORT_NODEMANAGER"
echo -e "\tOMSconsole port found at $OMSHOST:$PORT_OMS"
echo -e "\tOMSproxy port found at $OMSHOST:$PORT_OMS_JAVA"
echo -e "\tOMSupload port found at $OMSHOST:$PORT_UPL"
echo -e "\tWLSadmin found at $OMSHOST:$PORT_ADMINSERVER"
echo
echo -e "\tRepository DB version=$REPOS_DB_VERSION SID=$REPOS_DB_SID host=$REPOS_DB_HOST"

if [[ $RUN_DB_CHECK -eq "1" ]]; then
echo -e "\tRepository DB on OMS server, will check patches/parameters in $REPOS_DB_HOME"
fi

#exit 0

echo -e "\n(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)"

echo -e "\n\t(1a) Forbid SSLv2 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl2
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl2
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl2
sslcheck BIPublisherOHS $OMSHOST $PORT_BIP_OHS ssl2
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl2
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl2
sslcheck OMSupload $OMSHOST $PORT_UPL ssl2
#sslcheck OPMN $OMSHOST $PORT_OPMN ssl2
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl2

echo -e "\n\t(1b) Forbid SSLv3 connections"
sslcheck Agent $OMSHOST $PORT_AGENT ssl3
sslcheck BIPublisher $OMSHOST $PORT_BIP ssl3
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER ssl3
sslcheck BIPublisherOHS $OMSHOST $PORT_BIP_OHS ssl3
sslcheck OMSconsole $OMSHOST $PORT_OMS ssl3
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA ssl3
sslcheck OMSupload $OMSHOST $PORT_UPL ssl3
#sslcheck OPMN $OMSHOST $PORT_OPMN ssl3
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER ssl3

echo -e "\n\t(1c) Permit TLSv1 connections"
sslcheck Agent $OMSHOST $PORT_AGENT tls1
sslcheck BIPublisher $OMSHOST $PORT_BIP tls1
sslcheck NodeManager $OMSHOST $PORT_NODEMANAGER tls1
sslcheck BIPublisherOHS $OMSHOST $PORT_BIP_OHS tls1
sslcheck OMSconsole $OMSHOST $PORT_OMS tls1
sslcheck OMSproxy $OMSHOST $PORT_OMS_JAVA tls1
sslcheck OMSupload $OMSHOST $PORT_UPL tls1
#sslcheck OPMN $OMSHOST $PORT_OPMN tls1
sslcheck WLSadmin $OMSHOST $PORT_ADMINSERVER tls1

echo -e "\n(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)"
ciphercheck Agent $OMSHOST $PORT_AGENT
ciphercheck BIPublisher $OMSHOST $PORT_BIP
ciphercheck NodeManager $OMSHOST $PORT_NODEMANAGER
ciphercheck BIPublisherOHS $OMSHOST $PORT_BIP_OHS
ciphercheck OMSconsole $OMSHOST $PORT_OMS
ciphercheck OMSproxy $OMSHOST $PORT_OMS_JAVA
ciphercheck OMSupload $OMSHOST $PORT_UPL
#ciphercheck OPMN $OMSHOST $PORT_OPMN
ciphercheck WLSadmin $OMSHOST $PORT_ADMINSERVER

echo -e "\n(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)"
certcheck Agent $OMSHOST $PORT_AGENT
democertcheck Agent $OMSHOST $PORT_AGENT
certcheck BIPublisher $OMSHOST $PORT_BIP
democertcheck BIPublisher $OMSHOST $PORT_BIP
certcheck NodeManager $OMSHOST $PORT_NODEMANAGER
democertcheck NodeManager $OMSHOST $PORT_NODEMANAGER
certcheck BIPublisherOHS $OMSHOST $PORT_BIP_OHS
democertcheck BIPublisherOHS $OMSHOST $PORT_BIP_OHS
certcheck OMSconsole $OMSHOST $PORT_OMS
democertcheck OMSconsole $OMSHOST $PORT_OMS
certcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
democertcheck OMSproxy $OMSHOST $PORT_OMS_JAVA
certcheck OMSupload $OMSHOST $PORT_UPL
democertcheck OMSupload $OMSHOST $PORT_UPL
#certcheck OPMN $OMSHOST $PORT_OPMN
#democertcheck OPMN $OMSHOST $PORT_OPMN
certcheck WLSadmin $OMSHOST $PORT_ADMINSERVER
democertcheck WLSadmin $OMSHOST $PORT_ADMINSERVER

echo -e "\n(4) Checking EM13c Oracle home patch levels against $PATCHDATE baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)"

if [[ $RUN_DB_CHECK -eq 1 ]]; then

if [[ "$REPOS_DB_VERSION" == "12.1.0.2.0" ]]; then
echo -ne "\n\t(4a) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) PSU 12.1.0.2.160119 (JAN2016) (21948354)... "
opatchcheck ReposDBHome $REPOS_DB_HOME 21948354

echo -ne "\n\t(4a) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) ORACLE JAVAVM COMPONENT 12.1.0.2.160119 DATABASE PSU (JAN2016) (22139226)... "
opatchcheck ReposDBHome $REPOS_DB_HOME 22139226
fi

echo -ne "\n\t(4b) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) sqlnet.ora SSL_VERSION parameter (1545816.1)... "
paramcheck SSL_VERSION $REPOS_DB_HOME sqlnet.ora

echo -ne "\n\t(4b) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... "
paramcheck SSL_CIPHER_SUITES $REPOS_DB_HOME sqlnet.ora

echo -ne "\n\t(4b) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) listener.ora SSL_VERSION parameter (1545816.1)... "
paramcheck SSL_VERSION $REPOS_DB_HOME listener.ora

echo -ne "\n\t(4b) OMS REPOSITORY DATABASE HOME ($REPOS_DB_HOME) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... "
paramcheck SSL_CIPHER_SUITES $REPOS_DB_HOME listener.ora
fi

echo -ne "\n\t(4c) OMS CHAINED AGENT HOME ($AGENT_HOME) EM-AGENT BUNDLE PATCH 13.1.0.0.160331 (22823268)... "
opatchcheck Agent $AGENT_HOME 22823268

echo -ne "\n\t(4c) OMS CHAINED AGENT HOME ($AGENT_HOME) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160331 MONITORING (22920712)... "
opatchcheck Agent $AGENT_HOME 22920712

echo -ne "\n\t(4c) OMS CHAINED AGENT HOME ($AGENT_HOME) EM FMW PLUGIN BUNDLE PATCH 13.1.1.0.160331 MONITORING (22936491)... "
opatchcheck Agent $AGENT_HOME 22936491

echo -ne "\n\t(4c) OMS CHAINED AGENT HOME ($AGENT_HOME) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160331 MONITORING (22823189)... "
opatchcheck Agent $AGENT_HOME 22823189

echo -ne "\n\t(4d) OMS HOME ($OMS_HOME) ENTERPRISE MANAGER FOR OMS PLUGINS 13.1.1.0.160331 (22920724)... "
omspatchercheck OMS $OMS_HOME 22920724

echo
echo

if [[ $FAIL_COUNT -gt "0" ]]; then
echo "Failed test count: $FAIL_COUNT - Review output"
test $VERBOSE_CHECKSEC -ge 1 && echo -e $FAIL_TESTS
else
echo "All tests succeeded."
fi

echo
echo "Visit https://pardydba.wordpress.com/2016/04/05/securing-oracle-enterprise-manager-13c/ for the latest version."
echo

exit

EM13c TLS Security Checkup Script Sample Output


Performing EM13c security checkup version 0.1 on omshost.domain.com at Wed Apr 6 11:23:54 EDT 2016.

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst/em/EMGC_OMS1/embip.properties

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9851
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK

(1c) Permit TLSv1 connections
Confirming tls1 available for Agent at omshost.domain.com:3872... OK
Confirming tls1 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1 available for BIPublisherOHS at omshost.domain.com:9851... OK
Confirming tls1 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1 available for WLSadmin at omshost.domain.com:7102... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 1477287.1, 1905314.1, 1067411.1)
Checking LOW strength ciphers on Agent (omshost.domain.com:3872)... OK
Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK

Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403)... OK
Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK

Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK
Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9851)... OK

Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK

Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK

Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903)... OK
Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK

Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... FAILED - PERMITS MEDIUM STRENGTH CIPHER CONNECTIONS
Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
Checking certificate at Agent (omshost.domain.com:3872)... FAILED - Found self-signed certificate
Checking demo certificate at Agent (omshost.domain.com:3872)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803)... OK
Checking certificate at NodeManager (omshost.domain.com:7403)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9851)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9851)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301)... OK
Checking certificate at OMSupload (omshost.domain.com:4903)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102)... OK

(4) Checking EM13c Oracle home patch levels against 5 Apr 2016 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) PSU 12.1.0.2.160119 (JAN2016) (21948354)... OK
Patch 21948354 : applied on Fri Jan 22 11:03:15 EST 2016 Patch description: "Database Patch Set Update : 12.1.0.2.160119 (21948354)"

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) ORACLE JAVAVM COMPONENT 12.1.0.2.160119 DATABASE PSU (JAN2016) (22139226)... OK
Patch 22139226 : applied on Fri Jan 22 10:58:00 EST 2016 19699946, 19176885, 19623450, 22139226, 19909862, 21811517, 19223010

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES128_CBC_SHA,SSL_RSA_WITH_AES256_CBC_SHA)

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM-AGENT BUNDLE PATCH 13.1.0.0.160331 (22823268)... OK
Patch 22823268 : applied on Wed Apr 06 09:53:14 EDT 2016 22823268, 22384903, 21779343, 22293781, 22568679, 22565184, 22988508

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM DB PLUGIN BUNDLE PATCH 13.1.1.0.160331 MONITORING (22920712)... OK
Patch 22920712 : applied on Wed Apr 06 10:40:33 EDT 2016 22920712, 22902094, 22908077

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM FMW PLUGIN BUNDLE PATCH 13.1.1.0.160331 MONITORING (22936491)... OK
Patch 22936491 : applied on Wed Apr 06 10:46:28 EDT 2016 22461699, 22936491

(4c) OMS CHAINED AGENT HOME (/oracle/oem/agent13cR1/agent_13.1.0.0.0) EM SI PLUGIN BUNDLE PATCH 13.1.1.0.160331 MONITORING (22823189)... OK
Patch 22823189 : applied on Wed Apr 06 11:07:32 EDT 2016 22128210, 22823189

(4d) OMS HOME (/oracle/oem/Middleware13cR1) ENTERPRISE MANAGER FOR OMS PLUGINS 13.1.1.0.160331 (22920724)... OK
oracle.sysman.emas.oms.plugin/13.1.1.0.0 Plugin 22920724 22823175 oracle.sysman.si.oms.plugin/13.1.1.0.0 Plugin 22920724 22589347 oracle.sysman.db.oms.plugin/13.1.1.0.0 Plugin 22920724 22823156

Failed test count: 8 - Review output

ciphercheck:Agent @ omshost.domain.com:3872:Permits MEDIUM strength ciphers
ciphercheck:BIPublisher @ omshost.domain.com:9803:Permits MEDIUM strength ciphers
ciphercheck:NodeManager @ omshost.domain.com:7403:Permits MEDIUM strength ciphers
ciphercheck:BIPublisherOHS @ omshost.domain.com:9851:Permits MEDIUM strength ciphers
ciphercheck:OMSproxy @ omshost.domain.com:7301:Permits MEDIUM strength ciphers
ciphercheck:OMSupload @ omshost.domain.com:4903:Permits MEDIUM strength ciphers
ciphercheck:WLSadmin @ omshost.domain.com:7102:Permits MEDIUM strength ciphers
certcheck:Agent @ omshost.domain.com:3872 found self-signed certificate

Visit https://pardydba.wordpress.com/2016/04/05/securing-oracle-enterprise-manager-13c/ for the latest version.

I have just released my whole genome sequence (WGS) to the public domain (CC0, no rights reserved), via the Harvard Personal Genome Project (PGP). I believe that my data represents both the first $1000 genome-with-analysis ever performed as well as the first $1000 genome released for public use. Thank you to both the PGP and to Veritas Genetics for making this possible. I would like to specifically thank Mirza Cifric, CEO of Veritas Genetics and also Christen Hart of Veritas for acting as my liaison and dealing with my frequent email requests for status updates. From my PGP profile page you can download my genome data (as a BAM file (17.8GB) or in VCF format (383MB)), as well as my 23andMe (v3, pre-FDA letter) SNP chip data and my full mitochondrial DNA sequence as tested by FamilyTreeDNA (since deposited in GenBank as accession ID KU530226).

Why would I do this?

Put simply, I wanted to make a contribution to science. Further, since working for a genomic drug development company in the 2000s where I met, then married, a bioinformatician, I’ve had an interest in the potential applications of genomics, from what some then referred to as the “pharmaceutically tractable genome” to today’s “precision medicine”. That employer spun off an early DNA sequencing platform (454 Life Sciences pyrosequencing, the first company to complete and make public an individual human genome), and I find it fitting that an ex-employee, and one from the IT staff, not even the scientific team, would release the first public $1000 genome.

I would like to see science make some good use of my genetic data. Only a relatively small number of whole genome sequences available for scientific research without privacy or intellectual property encumbrances exist. As a participant in the PGP, by making my genome available I hope not only to directly support scientific research but to aid the PGP’s other research goal to identify the risk and consequences of having one’s genetic data available to the public without any effort at de-identification or obfuscation. I have the benefit of living in one of the few states with genetic information laws that exceed the US Federal Genetic Information Nondiscrimination Act in placing restrictions on life insurance providers and others.

After my first blood labs with my current primary care doctor, she told me that I had the absolute worst blood levels of vitamin D that she had ever seen, along with the best HDL/LDL cholesterol levels she had seen. This comes from a genetic basis, not anything that I have pursued through diet or lifestyle. In fact my cholesterol should be, frankly, terrible, and though I live only a few miles south of the 45th parallel I get enough sun that lack of exposure can’t account for my vitamin D levels alone. My 23andMe data, when run through Promethease, reveals a train wreck throughput the vitamin D pathway, as well as matching many variants known to increase HDL cholesterol. With my whole genome sequence released for any imaginable use, I hope that researchers can either spot something unique enough on its own or work my data into genome wide association studies (GWAS) to tease out some drug targets or relevant alleles.

As a PGP participant I have filled out the PGP’s phenotype surveys to help associate phenotypes with my genotype. I have done the same at OpenHumans and remain willing to provide further phenotype data on request. I will attend the GET Conference and GET Labs 2016 at the end of April and get signed up with some other research studies.

You can also find my autosomal SNP chip data on GEDMatch as kit M205442, my YDNA data at ysearch under id CZVXU, and my full mitochondrial DNA sequence in GenBank as KU530226 (though services report my mtDNA haplogroup as U2e1*, I hope the next build of PhyloTree will note the mtDNA SNPs I carry extraneous to U2e1 and define a new haplogroup as with my deposition several mtDNA sequence motifs now have three independent depositions, enough to justify naming a new U2e1* branch). I have much of my genealogy traced several generations back and several apparent triangulation groups worth of matches. Genealogy traces my surname back to the Paradis in Quebec but hits a brick wall in the mid 1800s, though my YDNA 67-STR results at FTDNA show close matches with other tested Paradis males who have traceable lineages back to Pierre Paradis of Mortagne-au-Perche, France (d. 1675), apparent patriarch of new world Paradis/Pardy lines. Several of my lines go back to early US colonials (Trowbridge provides my nexus to Charlemagne, though I’ve found no Mayflower descendents), as well as mixed ancestry (French/German/more) Creoles along the German Coast in Louisiana. I also have a bit of direct Scottish (Halcro) ancestry along with other Scots-Irish.

How can a security and privacy aware individual choose to release this data?

For me, the recognition that sequencing continues to fall in price and will eventually become ubiquitous to the point of banality, coupled with the fact that we shed DNA all day long convinces me that any genetic privacy we may believe we have now exists only for a disappearing moment in history and only in lieu of a determined adversary willing to put some effort into collection. Setting aside the issue of disclosing one’s unique genetic signature to third parties, simply knowing what secrets sit in one’s own DNA empowers some individuals but makes others uneasy. Some people do not want to know if their genetics give them a high probability of Alzheimers, or a disposition to cancer. Some regulators believe they cannot trust the public to make responsible decisions once given knowledge of the forbidden fruit in their genetic code. Because science does not yet know enough about the complex interactions of all parts of the genome to determine the exact medical significance of every gene or non-gene variant, the interpretation of your static genome can and will change with the ongoing discovery of new genetic associations and with failures to replicate previously reported associations. By donating my sequence to an unencumbered public dataset I hope to help speed up this process and embolden others to take this step to share for science, with eyes wide open as to the limitations of data de-identification and possibilities of personalized medicine. Whether you share your genome through the PGP, your microbiome through uBiome, the next virus you catch through GoViral, your FitBit data through OpenHumans, your direct to consumer SNP chip results through OpenSNP, or any other data through any other platform, each of us has a unique chance to contribute to research to better lives today and our species tomorrow.

What does whole genome sequencing give a non-expert that SNP genotyping doesn’t?

Several years ago I took 23andMe’s genotyping test. As this occurred prior to the FDA sending 23andMe a nastygram barring them from reporting health-relevant results, I received a decent amount of information relevant to health issues. So why bother having a whole genome sequence done? To put it simply, a WGS has more long-term value than a genotyping SNP chip. As 23andMe V2 customers discovered, as time moves on and science learns more about genetic variants, and as new builds of the human genome get released, SNP results based on older data lose their relevance. New genome scaffolds obsolete what we believed we knew about older SNPs. New SNPs get discovered with more meaningful disease associations than those believed to associate with diseases years ago during chip design. With my whole genome sequence in my pocket, I have better positioning for the future as I can look up newly-reported variants going forward whether or not the designer of the probes on a SNP chip foresaw the relevance of that genetic region. If I develop cancer in the future, I or my medical providers can compare the sequence of a tumor cell to my genome sequence, easing the process of identifying genes that may have gone haywire and caused cancer, and potentially informing the selection of anti-cancer drugs that could save my life. Further, by ordering and releasing my whole genome sequence, scientists working with public datasets can perform more useful analyses than those available simply from releasing my SNP chip data.

Go use my data!

Updates

Mike Cariaso has graciously run Promethease against my WGS data. Results here.

In this post, I will provide my review of Full Genomes Corp‘s service offering third party analysis of raw data produced by Veritas Genetics‘ $999 whole genome sequencing (Veritas myGenome). After I released my raw genome data to the public domain, FGC contacted me and offered to run my WGS data through their BAM processing pipeline at no cost. I naturally accepted and agreed to write a review.

This service from FGC includes three categories of analysis: mtDNA, YDNA, and autosomal ancestry. As of now, I have received my mtDNA and YDNA results; the autosomal analysis takes longer to produce and I will leave it out of scope for this review.

Getting Started

After creating an account on the FGC site, I needed to provide them with access to the BAM file that Veritas Genetics produced. My participation in the Personal Genome Project made this easy as I only had to give them the URL to my BAM file on the PGP public data repository.

A little bit more than two weeks later I received email reporting that I had results ready. When I logged back in to FGC a prominent link provided access to download all of my results in a single zip archive. This zip archive contained a readme file directing me to two PDF documents with further information: one focused on extracting private SNPs from YDNA results and the second describing the individual data files FGC returns, which I will get to below.

Mitochondrial DNA results

I have already had my full mitochondrial DNA sequenced by FamilyTreeDNA, so I did not expect to learn anything new from FGC’s data analysis, which produced two files. The first file contains a list of variants found in my mtDNA with respect to the Yoruba reference sequence by position. The second file contains my full mtDNA sequence in FASTA format.

The FASTA file took me by surprise, as they indicated a heteroplasmic length variant that FamilyTreeDNA had not come across (or had not informed me of) in their Sanger sequencing. FGC found a deletion at position 310, the loss of a T flanked by C repeats on both sides. I do not know if this information will turn out relevant for me, but who knows, I prefer to have it.

YDNA results

FGC grouped my YDNA results into two folders: YSTR and YSNP.

YSTR

YSTR results consisted of two output files generated from lobSTR. The first file contains roughly 3000 lines of data reporting identified YSTRs according to NIST/lobSTR standards, with some additional markers FGC has added to lobSTR.

The second file contains a subset of the first file including only those YSTR markers which FamilyTreeDNA tests and reports, counted according to FamilyTreeDNA’s standards. Mine reported values for 95 FTDNA-style markers.

Prior to whole genome sequencing I had only FTDNA’s 67 marker YSTR results combined with 23andMe‘s v3 chip Y SNPs with which to determine my YDNA haplogroup, giving nothing more specific than the huge R1b M269 group. I have not yet found my YSTR results from FGC particularly useful as not very many males from my line appear to have taken YDNA testing, so I do not have many data points to compare to.  I do have several close matches on FTDNA’s 67 marker test sharing variants of my surname which have convinced me that I don’t need to consider non paternity events along my direct male line going back at least 400 years based on the known years when Paradis YDNA arrived in Canada from France.

Once more Paradis-descended men take YDNA tests like the Veritas myGenome, FGC Y-Elite, FTDNA Big Y or others, I expect this data to have more value in tracing drift across this line.

YSNP

YSNP results consisted of five separate files. Two described as variant discovery reports, two as variant genotyping, and one haplogroup classification report containing output from yKnot that identifies my sample’s place in the ISOGG tree.

Haplogroup Classification

I have provided below a portion of my yKnot file showing the placement of my YDNA on the ISOGG tree back to the R1b M343+ branch. For the moment, I sit on the S1217+/Z295+ branch (ISOGG, Big Tree). I do not match any subclades of S1217+/Z295+ yet identified, but I will follow developments in this area, and, having my genome already sequenced, can place myself on future revised trees without the need for any further SNP testing.

*Extras: Z1518+, Y4010+, 50f2(P)+, Z14907+, PH3244*, Y2550+, P80+, CTS1789+, CTS12019+, L1228+, M3629+, Z3327+, Z28+, FGC5628+, CTS12440+, PF2372+, M162_1*, FGC5085+, Z13028+, P266+, Z12253+, L798+, DYS257_2+, Z28771*, P27.2_2+, Y2252+, CTS616+, CTS2646*, M118+, M236+, Y2754+, FGC20667*, M141+, L665+, L588+, Z14350+, P34_5+, Z6859+, Z889+, Z13537*, Z6171+, Z1237+, FGC756+, BY451+,     P19_1*, P79*, PF2276+, Z16986+, M5220+, FGC1920+, Z12467+, Z1842+, V161.1+, V190+, CTS6911+, CTS2518+, FGC4872+, Y5185*, Y2986+, Z1101+, CTS32+, Z15165+, IMS-JST022457+, PF2779+, S730+, S504+, Z836*, Z14050+, IMS-JST029149+, M1994*, L990+, P198+, Z16208+, PF3126+, Z2182*
R1b1a2a1a2a1a1a
|Matches: S1217+, Z295+
|____R1b1a2a1a2a1a1
     |Matches: S230+, Z209+, S356+, Z220+
     |____R1b1a2a1a2a1a
          |Matches: Z272+
          |*No-calls: Z274?, S229?
          |____R1b1a2a1a2a1
               |Matches: Z195+, S227+
               |*No-calls: S355?, Z196?
               |____R1b1a2a1a2a
                    |Matches: DF27+, S250+
                    |____R1b1a2a1a2
                         |Matches: P312+, PF6547+, S116+
                         |____R1b1a2a1a
                              |Matches: L151+, PF6542+, L52+, PF6541+, P310+, PF6546+, S129+, P311+, PF6545+, S128+, PF6539+
                              |*No-calls: (being investigated as to placement: L11?, S127)?
                              |____R1b1a2a1
                                   |Matches: L51+, M412+, PF6536+, S167+
                                   |____R1b1a2a
                                        |Matches: L23+, PF6534+, S141+, L49.1+, S349.1+
                                        |____R1b1a2
                                             |Matches: M269+, CTS623+, CTS2664+, PF6454+, CTS3575+, PF6457+, CTS8728+, L1063+, PF6480+, S13+, CTS12478+, PF6529+, F1794+, PF6455+, L265+, PF6431+, L407+, PF6252+, L478+, PF6403+, L482+, PF6427+, L483+, L500+,   PF6481+, L773+, PF6421+, YSC0000276+, L1353+, PF6489+, YSC0000294+, M520+, PF6410+, PF6399+, S10+, PF6404+, PF6505+, YSC0000225+,   PF6409+, PF6411+, PF6425+, PF6430+, PF6432+, PF6434+, PF6438+, PF6475+, S17+, YSC0000269+, PF6482+, YSC0000203+, PF6485+, S3+, PF6494+, PF6495+, PF6497+, YSC0000219+, PF6500+, PF6507+, PF6509+, L150.1+, PF6274.1+, S351.1+
                                             |*No-calls: PF6443?
                                             |**Mismatches: CTS8591- (exp. +), CTS8665- (exp. +), FGC464- (exp. +), CTS10834- (exp. +), CTS11468- (exp. +), FGC49- (exp. +)
                                             |____R1b1a
                                                  |Matches: P297+, PF6398+, L320+
                                                  |____R1b1
                                                       |Matches: P25_3+, L278+, M415+, PF6251+
                                                       |**Mismatches: P25_1- (exp. +), P25_2- (exp. +)
                                                       |____R1b
                                                            |Matches: M343+, PF6242+

Variant Genotyping

The first variant genotyping file provides my results at a little over 54,000 known SNPs. The second variant genotyping file provides results for an additional 16,600 SNPs. The results provided include counts of each base called at the SNP position as identified in my BAM file data, the SNP position on the chromosome, and the build 37 reference sequence call at that position. I do not know the criteria used to place each SNP in each file. I consider these files more as an intermediate step in the data analysis, used to generate the other returned files, but I expect I will find some more direct use for them as well.

Variant Discovery

The two variant discovery reports provide the most detailed and useful information in my opinion, as they include quality rankings on variants as well as the specific details of variants such as SNPs and INDELs. Even more usefully, these files contain the results for the kits most similar to mine within FGC’s database, which can help in identifying private variants that originated in much more recent genealogical times. Because these files include data from others as well as my own, I cannot comfortably release them to the general public without redacting other individuals’ data. For public facing purposes if someone wanted to run comparisons against my detailed data I would most likely refer them to the Big Tree (if R1b) or advise that they pursue their own analysis with FGC directly. The how-to document FGC provides with this analysis (Reading the Full Genomes analysis reports) explains working with this data much better than I could in my own words. The inclusion of quality scores greatly simplifies the process of narrowing down on key SNPs, and I look forward to spending more time with this data — probably after more Paradis males have had next generation YDNA sequencing as my results appear rather distant from the nearest matching males in any database except for the one Paradis I’ve found with a Big-Y at FTDNA.

Data Sharing

It pleased me to see that FGC offers a very quick and easy method to share your results with any email address you provide. I took advantage of this to share my data with Alex Williamson for inclusion in the Big Tree to aid in reconstructing the phylogeny of the R1b tree under R P312. For now, my Big Tree entry sits in the R-Z295/S1217 paragroup, awaiting more submissions sharing SNPs with me to help identify a terminal SNP more recent than the estimated 3900 year old Z295. I don’t match any SNPs identified as downstream of Z295 on the FTDNA tree, the ISOGG tree, or the YFull tree. I encourage any other Z295 or Paradis/Pardy/Paradee/etc male to get your YDNA analyzed and shared with these projects so we can better place ourselves on the tree.

More Info

If this has interested you, I highly recommend you take a look at another review and description of FGC’s analysis.

Oracle recently released patch 23054246 (DATABASE PATCH SET UPDATE 12.1.0.2.160719) for database 12.1.0.2, containing security updates from the July 2016 critical patch update advisory.

[EDIT 20160726: Oracle has documented this issue in MOS note 2163593.1]

This patch appears to have introduced a dependency on libodbcinst. During my first attempt to install this patch, I received errors while linking libsqora. The error appears as follows in OPatch logs:


[Jul 20, 2016 11:22:57 AM] The following warnings have occurred during OPatch execution:
[Jul 20, 2016 11:22:57 AM] 1) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/oracle/oem/product/12.1.0/awrdb"....'/usr/lib64/gcc/x86_64-suse-linux/4.3/../../../../x86_64-suse-linux/bin/ld: cannot find -lodbcinst
collect2: ld returned 1 exit status
make: *** [/oracle/oem/product/12.1.0/awrdb/odbc/lib/libsqora.so.12.1] Error 1
'
[Jul 20, 2016 11:22:57 AM] 2) OUI-67124:Re-link fails on target "isqora".
[Jul 20, 2016 11:22:57 AM] 3) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/oracle/oem/product/12.1.0/awrdb"....'/usr/lib64/gcc/x86_64-suse-linux/4.3/../../../../x86_64-suse-linux/bin/ld: cannot find -lodbcinst
collect2: ld returned 1 exit status
make: *** [/oracle/oem/product/12.1.0/awrdb/odbc/lib/libsqora.so.12.1] Error 1
'
[Jul 20, 2016 11:22:57 AM] 4) OUI-67124:
NApply was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
"/oracle/oem/product/12.1.0/awrdb/.patch_storage/NApply/2016-07-20_11-20-22AM"


After installing the unixODBC package on my SLES11 system, this error went away.

[Update: see also Brian Peasland’s blog post “July 2016 PSU fails to make isqora” for a different workaround to this issue that does not involving installing any additional packages.]

At the time of release, Oracle’s installation requirements for database 12.1.0.2 listed the unixODBC package as an optional dependency, required only “[i]f you intend to use ODBC”. This no longer seems to hold true. At the moment Oracle has not made it clear whether or not patch 23054246 contains a bug that introduces the libodbcinst dependency or if the database platform will require this library in all cases going forward.

If you have attempted patch application without libodbcinst available, the opatch apply step will fail and you will have to manually revert the patch, following the instructions that OPatch provides and/or contact Oracle Support for guidance. In my case, I followed the instructions to revert, installed unixODBC, then attempted again to apply the patch, at which point it completed successfully as expected. If you have not yet attempted to apply this patch, I highly recommend installing unixODBC first. I have already seen two others report on Twitter that they encountered this issue but none have yet confirmed to me that installing unixODBC resolved the problem. I believe it will.

UPDATE: See also “BUG 24332805 – OUI-67124:RE-LINK FAILS ON TARGET “ISQORA” DURING JUL 2016 PSU APPLY” once made public.

For the first time in the 20 years I’ve worked with Oracle’s products, I will attend OpenWorld this year. If you see me there please feel free to stop me and say hello. I will attend sessions here and there, though I will not give any presentations or talks. I do expect to have an interesting surprise to share, though. Stay tuned.

I could not find any obvious documentation about locking down Oracle Enterprise Manager 13c management agents to forbid TLSv1 and TLSv1.1, permitting only TLSv1.2, so I went looking and found the emdpropdefs.xml file in $AGENT_HOME/agent_13.1.0.0.0/sysman/admin/ that documents the existence of the minimumTLSVersion property in emd.properties:

name='minimumTLSVersion'
modifiable='true'
defaultValue='TLSv1'
description='The oldest version of the TLS protocol which this agent should support when accepting connections or initiating connections to the OMS. Currently supported values are "TLSv1", "TLSv1.1", and "TLSv1.2".'
valueType='String'
advanced='true'
migrate='source'
filename='emd.properties'
category='Runtime Settings'
internal='true'
restartRequired='true'

I tested this parameter on my OMS server agent, restarted the agent, and confirmed with my Securing Oracle Enterprise Manager 13c script that the agent no longer allowed connections using any protocol other than TLSv1.2. Next I wanted to automated this, to avoid the effort of manually changing this property on each agent and then restarting that agent, so I went directly to EMCLI which allows EM13c admins to (among many other things) set agent properties and restart agents. I then created a script to fetch a list of all agents, check for the TLS protocols each agent permits, and then apply the change and restart the agent for every agent that I had not already locked down. I have copied this script below.

Before using the script, you must login to EMCLI using “emcli login -username=yourusername” and provide your password. For security reasons I elected not to wrap the EMCLI login within this script; that way you do not have to trust my script to handle your password securely, as the script never sees your password. For the step to restart your agents to work correctly, you need to make sure that your EM13c user account has preferred host credentials set for your agent targets that can successfully login to the host server and restart the agent.

Here is a copy of the script, followed by the (anonymized) output from a sample run. Someday soon I will get set up on github to make it easier to retrieve my scripts, but for now you can copy and paste this. This script expects to find the emcli binary inside of the $MW_HOME/bin directory, so make sure you have $MW_HOME set before running it, or provide the full path to EMCLI within the script. It will also log you out of EMCLI when the script completes.


#!/bin/bash
#
# This script will retrieve a list of agents from your EM13c environment,
# determine if they allow connections using TLS protocol versions older
# than TLSv1.2, and then disable all protocols older than TLSv1.2.
#
# Finally it will restart each modified agent to apply the change.
#
# You need to login to EMCLI first before running this script.
#
# Released v0.1: Initial beta release 5 Oct 2016
#
#
# From: @BrianPardy on Twitter
# https://pardydba.wordpress.com/
#
# Known functional on Linux x86-64, may work on Solaris and AIX.

EMCLI=$MW_HOME/bin/emcli

if [[ -x "/usr/sfw/bin/gegrep" ]]; then
GREP=/usr/sfw/bin/gegrep
else
GREP=`which grep`
fi

OPENSSL=`which openssl`

if [[ -x "/usr/bin/openssl1" && -f "/etc/SuSE-release" ]]; then
OPENSSL=`which openssl1`
fi

OPENSSL_HAS_TLS1_2=`$OPENSSL s_client help 2>&1 | $GREP -c tls1_2`

$EMCLI sync
NOT_LOGGED_IN=$?

if [[ $NOT_LOGGED_IN > 0 ]]; then
echo "Login to EMCLI with \"$EMCLI login -username=USER\" then run this script again"
exit 1
fi

for agent in `$EMCLI get_targets -targets=oracle_emd | grep oracle_emd | awk '{print $4}'`
do
echo
if [[ $OPENSSL_HAS_TLS1_2 > 0 ]]; then
echo -n "Checking TLSv1 on $agent... "

OPENSSL_RETURN=`echo Q | $OPENSSL s_client -prexit -connect $agent -tls1 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_RETURN == 0 ]]; then
echo "allows TLSv1"
else
echo "already forbids TLSv1"
fi
fi

if [[ $OPENSSL_HAS_TLS1_2 > 0 ]]; then
echo -n "Checking TLSv1.1 on $agent... "

OPENSSL_TLS11_RETURN=`echo Q | $OPENSSL s_client -prexit -connect $agent -tls1_1 2>&1 | $GREP Cipher | $GREP -c 0000`

if [[ $OPENSSL_RETURN == 0 ]]; then
echo "allows TLSv1.1"
else
echo "already forbids TLSv1.1"
fi
fi

if [[ $OPENSSL_RETURN == 0 || $OPENSSL_TLS11_RETURN == 0 ]]; then
$EMCLI set_agent_property -agent_name=$agent -name=minimumTLSVersion -value=TLSv1.2 -new

echo
echo "Restarting $agent to apply changes"
$EMCLI restart_agent -agent_name=$agent -credential_setname="HostCreds"
RESTART_RETURN=$?

if [[ $RESTART_RETURN != 0 ]]; then
echo "Unable to restart agent: restart agent manually or set preferred host credentials for agent"
fi
fi
done

$EMCLI logout

exit 0

Sample (anonymized) output below. Note how the script cannot restart an agent lacking preferred host credentials. In this case, I assign preferred host credentials and then re-run the script to complete the process.


Synchronized successfully

Checking TLSv1 on server1.subdomain.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server1.subdomain.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server2.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server2.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server3.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server3.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server4.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server4.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server5.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server5.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server6.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server6.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server7.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server7.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server8.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server8.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server9.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server9.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server10.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server10.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server11.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server11.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on server12.domain.com:1830... already forbids TLSv1
Checking TLSv1.1 on server12.domain.com:1830... already forbids TLSv1.1

Checking TLSv1 on omshost.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on omshost.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server13.domain.com:3872... allows TLSv1
Checking TLSv1.1 on server13.domain.com:3872... allows TLSv1.1
Agent Property minimumTLSVersion has been successfully updated to the value TLSv1.2.

Restarting server13.domain.com:3872 to apply changes
The Restart operation is in progress for the Agent: server13.domain.com:3872
The Agent "server13.domain.com:3872" has been restarted successfully.
---------------------
Operation Output
---------------------
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.Stopping agent ... stopped.Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.Starting agent ................ started.

Checking TLSv1 on server14.domain.com:1830... allows TLSv1
Checking TLSv1.1 on server14.domain.com:1830... allows TLSv1.1
Agent Property minimumTLSVersion has been successfully updated to the value TLSv1.2.

Restarting server14.domain.com:1830 to apply changes
The Restart operation is in progress for the Agent: server14.domain.com:1830
Unable to restart agent: restart agent manually or set preferred host credentials for agent

Checking TLSv1 on server15.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server15.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server16.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server16.domain.com:3872... already forbids TLSv1.1

Checking TLSv1 on server17.domain.com:3872... already forbids TLSv1
Checking TLSv1.1 on server17.domain.com:3872... already forbids TLSv1.1
Logout successful

I ran into the following issue while attempting to upgrade to the Oracle management agent on my one Windows (2008 R2, x64) server to the 13.2.0.0.0 version distributed with Oracle Enterprise Manager 13.2. The agent upgrade repeatedly failed in the “Upgrading Management Agent” step with an error message:


Exit Code :0
The version is 13.1.0.0.0
Checking for the version 13.2.0.0.0
The agent is not upgraded successfully
[...]
Plugins upgrade failed.
Plugin upgrade failed.
Check the file E:/agent13c/agent_inst/install/plugins_upgrade.txt.status on agent for plugin upgrade status.
Check latest E:/agent13c/agent_inst/install/logs/agentplugindeploy_.log
Plugins upgrade failed.0


The log file referenced in the error message contains some, but not much, additional information:


The command executed for discovery at _2016_10_12_11_48_28 is : E:\agent13c\agent_inst\bin\emctl_upgrade.bat set_discovery_root plugin oracle.sysman.si E:\agent13c\agent_13.2.0.0.0\plugins\oracle.sysman.si.discovery.plugin_13.2.1.0.0 13.2.1.0.0
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
return value is : 65280
Install case : set_discovery_root failure...existing on error value 65280


I fixed this issue by installing the Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package MFC Security Update package on the server. Finally, create a new directory called “prerelogs” inside of the …\agent_13.2.0.0.0\cfgtoollogs\ directory, or else my deployment hung forever during the “Performing prerequisite checks” step. Note that “prerelogs” does not contain the letter q, do not use ‘prereqlogs’. The agent deployment uses this directory to extract software for running the prerequisite checks; possibly setting SCRATCHPATH or another extra parameter would resolve this issue without manually creating the directory, but this worked for me.

After installation of the VC++ 2010 package, which did not require a reboot, followed by creating of the prerelogs directory, I repeated the agent upgrade and it completed successfully.

This issue may or may not occur on other Windows versions, I have not heard any other reports of problems. The first release of EM13c did not seem to require this package.

Oracle released Oracle Enterprise Manager 13cR2 at the beginning of October 2016. I have upgraded my production system to this new version, and here I provide a 13cR2-compatible version of my EM13c security checkup script. In addition to updating the script for EM13cR2, I have also updated it to take account of Oracle’s recommendation that single-instance non-RAC databases such as OEM repositories should now apply the DBBP Bundle Patch (previously known as the engineered systems bundle patch).

Latest Updates

Initial release on October 28, 2016. Other than repository database patches, only the WebLogic component in EM13cR2 has patches so far. This release includes the WLS PSU 12.1.3.0.161019.

Download

I have created a repository on github for my EM13c scripts. You can access it at https://github.com/brianpardy/em13c. To directly access the EM13cR2 security checkup script, use https://raw.githubusercontent.com/brianpardy/em13c/master/checksec13R2.sh.

Example Output


Performing EM13c R2 security checkup version 1.0 on omshost.domain.com at Fri Oct 28 12:11:25 EDT 2016.

Using port definitions from configuration files
/etc/oragchomelist
/oracle/oem/gc_inst1/em/EMGC_OMS1/emgc.properties
/oracle/oem/gc_inst1/em/EMGC_OMS1/embip.properties

Agent port found at omshost.domain.com:3872
BIPublisher port found at omshost.domain.com:9803
BIPublisherOHS port found at omshost.domain.com:9852
NodeManager port found at omshost.domain.com:7403
OMSconsole port found at omshost.domain.com:7802
OMSproxy port found at omshost.domain.com:7301
OMSupload port found at omshost.domain.com:4903
WLSadmin found at omshost.domain.com:7102

Repository DB version=12.1.0.2.0 SID=oemdb host=omshost.domain.com

Using OPENSSL=/usr/bin/openssl1 (has TLS1_2=2)
Repository DB on OMS server, will check patches/parameters in /oracle/oem/product/12.1.0/db

(1) Checking SSL/TLS configuration (see notes 1602983.1, 1477287.1, 1905314.1)

(1a) Forbid SSLv2 connections
Confirming ssl2 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl2 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl2 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl2 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming ssl2 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl2 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl2 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl2 disabled for WLSadmin at omshost.domain.com:7102... OK

(1b) Forbid SSLv3 connections
Confirming ssl3 disabled for Agent at omshost.domain.com:3872... OK
Confirming ssl3 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming ssl3 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming ssl3 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming ssl3 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming ssl3 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming ssl3 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming ssl3 disabled for WLSadmin at omshost.domain.com:7102... OK

(1c) Forbid TLSv1 connections
Confirming tls1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming tls1 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming tls1 disabled for WLSadmin at omshost.domain.com:7102... OK

(1c) Forbid TLSv1.1 connections
Confirming tls1_1 disabled for Agent at omshost.domain.com:3872... OK
Confirming tls1_1 disabled for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_1 disabled for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_1 disabled for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1_1 disabled for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_1 disabled for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_1 disabled for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_1 disabled for WLSadmin at omshost.domain.com:7102... OK

(1c) Permit TLSv1.2 connections
Confirming tls1_2 available for Agent at omshost.domain.com:3872... OK
Confirming tls1_2 available for BIPublisher at omshost.domain.com:9803... OK
Confirming tls1_2 available for NodeManager at omshost.domain.com:7403... OK
Confirming tls1_2 available for BIPublisherOHS at omshost.domain.com:9852... OK
Confirming tls1_2 available for OMSconsole at omshost.domain.com:7802... OK
Confirming tls1_2 available for OMSproxy at omshost.domain.com:7301... OK
Confirming tls1_2 available for OMSupload at omshost.domain.com:4903... OK
Confirming tls1_2 available for WLSadmin at omshost.domain.com:7102... OK

(2) Checking supported ciphers at SSL/TLS endpoints (see notes 2138391.1, 1067411.1)
Checking LOW strength ciphers on Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on Agent (omshost.domain.com:3872)... OK
Checking HIGH strength ciphers on Agent (omshost.domain.com:3872)... OK

Checking LOW strength ciphers on BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisher (omshost.domain.com:9803)... OK
Checking HIGH strength ciphers on BIPublisher (omshost.domain.com:9803)... OK

Checking LOW strength ciphers on NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on NodeManager (omshost.domain.com:7403)... OK
Checking HIGH strength ciphers on NodeManager (omshost.domain.com:7403)... OK

Checking LOW strength ciphers on BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on BIPublisherOHS (omshost.domain.com:9852)... OK
Checking HIGH strength ciphers on BIPublisherOHS (omshost.domain.com:9852)... OK

Checking LOW strength ciphers on OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSconsole (omshost.domain.com:7802)... OK
Checking HIGH strength ciphers on OMSconsole (omshost.domain.com:7802)... OK

Checking LOW strength ciphers on OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSproxy (omshost.domain.com:7301)... OK
Checking HIGH strength ciphers on OMSproxy (omshost.domain.com:7301)... OK

Checking LOW strength ciphers on OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on OMSupload (omshost.domain.com:4903)... OK
Checking HIGH strength ciphers on OMSupload (omshost.domain.com:4903)... OK

Checking LOW strength ciphers on WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking MEDIUM strength ciphers on WLSadmin (omshost.domain.com:7102)... OK
Checking HIGH strength ciphers on WLSadmin (omshost.domain.com:7102)... OK

(3) Checking self-signed and demonstration certificates at SSL/TLS endpoints (see notes 1367988.1, 1399293.1, 1593183.1, 1527874.1, 123033.1, 1937457.1)
Checking certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking demo certificate at Agent (omshost.domain.com:3872, protocol tls1_2)... OK
Checking certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking demo certificate at BIPublisher (omshost.domain.com:9803, protocol tls1_2)... OK
Checking certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking demo certificate at NodeManager (omshost.domain.com:7403, protocol tls1_2)... OK
Checking certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking demo certificate at BIPublisherOHS (omshost.domain.com:9852, protocol tls1_2)... OK
Checking certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking demo certificate at OMSconsole (omshost.domain.com:7802, protocol tls1_2)... OK
Checking certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking demo certificate at OMSproxy (omshost.domain.com:7301, protocol tls1_2)... OK
Checking certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking demo certificate at OMSupload (omshost.domain.com:4903, protocol tls1_2)... OK
Checking certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK
Checking demo certificate at WLSadmin (omshost.domain.com:7102, protocol tls1_2)... OK

(4) Checking EM13c Oracle home patch levels against 19 Oct 2016 baseline (see notes 1664074.1, 1900943.1, 822485.1, 1470197.1, 1967243.1)

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) DATABASE BUNDLE PATCH: 12.1.0.2.161018 (OCT2016) (24340679)... OK
24340679 24340679 Patch 24340679 : applied on Tue Oct 25 11:45:36 EDT 2016 Patch description: "DATABASE BUNDLE PATCH: 12.1.0.2.161018 (24340679)"

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) Database PSU 12.1.0.2.161018, Oracle JavaVM Component (OCT2016) (24315824)... OK
Patch 24315824 : applied on Tue Oct 25 11:55:55 EDT 2016 19231857, 19895362, 23265965, 24448282, 22670413, 24315824, 19623450

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) OCW Interim patch for 24846605 (24846605)... OK
Patch 24846605 : applied on Tue Oct 25 11:51:12 EDT 2016 Patch description: "OCW Interim patch for 24846605"

(4a) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) EM QUERY WITH SQL_ID 4RQ83FNXTF39U PERFORMS POORLY ON ORACLE 12C RELATIVE TO 11G (20243268)... OK
Patch 20243268 : applied on Tue Oct 25 14:30:20 EDT 2016 20243268

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_SERVER parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(AES128,AES256,AES192,3DES168)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.ENCRYPTION_CLIENT parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER parameter (76629.1, 2167682.1)... OK
(SHA1)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_SERVER parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_TYPES_CLIENT parameter (76629.1, 2167682.1)... OK
(SHA1)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SQLNET.CRYPTO_CHECKSUM_CLIENT parameter (76629.1, 2167682.1)... OK
requested

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) sqlnet.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_VERSION parameter (1545816.1)... OK
1.0

(4b) OMS REPOSITORY DATABASE HOME (/oracle/oem/product/12.1.0/db) listener.ora SSL_CIPHER_SUITES parameter (1545816.1)... OK
(SSL_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA)

(4c) *NEW* (/oracle/oem/Middleware13cR2) WLS PATCH SET UPDATE 12.1.3.0.161019 (23744018)... OK
Patch 23744018 : applied on Wed Oct 19 09:56:54 EDT 2016

(5) Checking EM13c Java patch levels against 19 Oct 2016 baseline (see notes 1492980.1, 1616397.1)

(5a) WLS (/oracle/oem/Middleware13cR2/oracle_common/jdk) JAVA SE JDK VERSION 1.7.0-111 (13079846)... OK
1.7.0_111

All tests succeeded.

Visit https://pardydba.wordpress.com/2016/10/28/securing-oracle-enterprise-manager-13cr2/ for the latest version.

Previous Versions

• Enterprise Manager 13c R1
• Enterprise Manager 12c R4

I have just noticed and resolved an issue in my EM13c R2 AWR Warehouse environment that I brought upon myself, hence a blog post for any others who might run into this, which also seems like a good time to release the scripts I use to generate and populate Oracle wallets for my EM13c agents.

After moving an AWRW source database from one EM13c managed server to a different EM13c managed server (same OS, same DB release), AWRW loads from that server began to fail. While debugging the issue, I first had to resolve an already-documented issue (see MOS note 2075341.1) where the source database had a NULL definition for the CAW_EXTR directory object, then fix up the data in the DBSNMP.CAW_EXTRACT_PROPERTIES table to reflect the CAW_EXTR directory. After resolving that, AWRW extracts ran successfully from the source database, but began to hang indefinitely during the CAW_RUN_ETL_NOW job in the transferAWR/transferFile job step, displaying only a cryptic error message:

A helpful error message

I ran through many debugging steps: changing preferred credentials, bouncing the agents, checking for firewalls blocking connectivity, none seemed to help. Eventually I realized the step I had missed in setting up the new managed server where the source database now runs: I had not generated an Oracle wallet for the agent on the new server, while I did have an Oracle wallet for the agent on the previous, now-retired server. This created an issue because I have secured the agent on my OMS host (where my AWRW repository database runs) with a custom third party certificate, and the new agent, lacking a wallet containing a trusted root certificate to which it could trace the repository agent’s certificate, could not initiate a connection from the AWRW source DB host agent to the AWRW repository DB host agent.

I generated a wallet for the new agent, added the trusted root certificate and a certificate for the host to the wallet, stopped the agent, deployed the wallet, and started the agent. After those steps, running the AWRW load from this source database completed successfully. I believe that the missing trusted root certificate prevented the creation of a secure channel between the two agents. I probably did not need to add the host certificate to resolve this problem, but consider it a good practice anyway.

If you read this far, you may find my create_agent_wallets.sh script useful to generate wallets and certificate signing requests for every agent in your environment. If you find the wallet creation script useful, you may also find my import_agent_wallets.sh script useful to populate those wallets with signed certificates received from your CA.