Integration Concepts

Amazon EC2 (Amazon Elastic Compute Cloud)?
Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.

Amazon S3?
Amazon Simple Storage Service is storage for the Internet. It is designed to make web-scale computing easier for developers.
Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

Amazon DynamoDB & Amazon RDS databases?

DynamoDB makes it simple and cost-effective to store and retrieve any amount of data, as well as serve any level of request traffic. All data items are stored on solid-state drives, which provide high I/O performance and can more efficiently handle high-scale requests. An AWS user interacts with the service by using the AWS Management Console or a DynamoDB API.

DynamoDB uses a NoSQL database model, which is nonrelational, allowing documents, graphs and columnar among its data models. A user stores data in DynamoDB tables, then interacts with it via GET and PUT queries, which are read and write operations, respectively.

Amazon CloudFront CDN?

Amazon CloudFront is a content delivery network (CDN) offered by Amazon Web Services. Content delivery networks provide a globally-distributed network of proxy servers which cache content, such as web videos or other bulky media, more locally to consumers, thus improving access speed for downloading the content.

Cloud vs Hybrid Cloud?
It's inclusion of Public and Private.

How will you stop a dos attack?

how will you troubleshoot a process in linux?

Answer was using, netstat, ps, lsof, gdb, top. the answer he was expecting was thread dump. this was troubleshooting httpd. you can only thread dump on java but i didn't want to argue with the guy, i just said I dont use that. He thinks he can do thread dump on any process obviously he never really troubleshooted any process in linux

What is error 500?

What is the difference between rdb and nosql?

how would you set up a server with db, no downtime, redundant, resilient, multi location, ..etc?

how will you troubleshoot issues on linux, like network, process..etc?

Lots of S.T.A.R. questions?

tell me a time when you have failed, achieved something more than you think?

stressed, ...etc how did you deal with it, what did you do...etc ?

when you enter a domain name into browser , what happens?

What is proxy?

What is the difference proxy and routing?

how CDN works?

What is IDS/IPS, which layer in osi does it work?

how would you do a sql replication?

What is NAT?

What is PAT?

how does load balancer works? which brand would you use? (I5)

how would you connect to your servers for administration and troubleshooting, linux and windows?

what are the differences between windows and linux?

what is error 500 on a web page?

esxi hosts?

VMWare Servers?

physical supermicro superservers?

Why HP?

The frustration was that in-person interview was mostly questions about what I like to do on weekends, what are my hobbies?

Design a fully redundant web solution in AWS?

Complicated Customer support and escalation scenarios?

How do you scale your 2 web and 1 DB app to 1000s of users and accept ordering system. Load Balancer, CDN, Storage?

Describe for us your process for working through a customer issue resolution cycle and your approach to customer communication?

SAN/NAS differences?

Troubleshooting in a "hybrid cloud" environments?

RAID basics?

My interest in cloud and any personal or professional experience there?

SQL/NoSQL, architectural questions?

Tell me about a time" questions, etc?

Technical questions - being to the point, architectural - discussed a lot, thought out loud, with pros and cons, multiple solutions and why. In behavior questions, emphasized on "I" and what I did as individual with STAR approach?

What is the difference between DoS and DDoS?
The Difference Between DoS and DDos Attacks. A Denial of Service (DoS) attack is different from a DDoS attack. The DoS attack typically uses one computer and one Internet connection to flood a targeted system or resource. The DDoS attack uses multiple computers and Internet connections to flood the targeted resource.

Networking (DNS, IPsec, BGP, VPN, Load Balancing)

Database (MySQL, SQL, Oracle)

Tools for Operations Automation and Configuration Management (e.g. Chef/Puppet)

Continuous Integration?

Wake up before your competition do!

Some inspiration from 5AM group on Facebook

You might not have a competitive advantage over your competition and maybe you don't have all the tools and resources to play with the big boys in your industry.

~

But one thing you do have is the ability to out work your competition!

~

You don't need to come up with anymore excuses about why you aren't where you want to be, what you need to do is get up earlier, show up and show the world you're not here to make up the numbers and be another statistic!

~

If you're not prepared to out work your competition and wake up before 5am every single day then maybe it's time to ask yourself how serious you are about your goals and dreams!

~

Stop doing normal things and stop performing at average levels because that is exactly why you aren't where you want to be...

~

Be unusual, be abnormal, be obsessed, be prepared to wake up at 4:00am and work harder than anyone you've ever met because that is the only way you're ever going to become exceptional, renowned, world class and successful.

~

You've tried normal and you know where that gets you so now it's time to try crazy, obsessive, intense and exceptional and once you commit to that path....your life will never be the same again!

Grab Known Host Key From Remote SFTP Server in OpenSSH Format

1. Get the host key by running this command: ssh-keyscan > /tmp/ssh_out.openssh

2. Convert the host key format by running this command: ssh-keygen -l -f /tmp/ssh_out.openssh

Watch "Webinar : Practical SOA for the Solution Architect" on YouTube

https://www.youtube.com/watch?v=1KXKppaOgtY&feature=youtu.be

Determine Which version of Windows Installer is installed

To determine which version of Windows Installer is installed on a computer, follow these steps:

1) Click Start, click Run, type %systemroot%\system32, and then click. OK.
2) Right-click Msi.dll, and then click. Properties.
3) Click the Version tab, and then note the File version number.

Watch "WSO2Con 2011 - Enterprise Integration Patterns: Past, Present and Future" on YouTube

https://www.youtube.com/shared?ci=Kb-UT1kqJ3Q

JMeter SSH Sampler

https://github.com/linkeshkanna/Jmeter.SSH.FTP.Request

Jmeter Custom Sampler to make SFTP Request

To Create SSH FTP Requests in JMeter, we need this custom Sampler. This contains two jar files. 1. jmeter-ssh-sampler-1.0.2-SNAPSHOT.jar 2. jsch-0.1.53.jar We can generate this by downloading the source and building it using Maven.

I just built it using maven and checked in the jar files here.

To install this in Jmeter, 1. Copy the jmeter-ssh-sampler-1.0.2-SNAPSHOT.jar to the "Jmeter/Lib/ext" directory. 2. Copy the "jsch-0.1.53.jar" to the "Jmeter/Lib" directory 3. Restart Jmeter.

I have also added a sample test to list of the directory contents in a Public SFTP Server.

Install 3rdParty JAR on IBM B2B Sterling Integrator

If we want to install external JARs (3rdParty JARs) on Sterling Integrator here are the steps,

1. Stop B2BI Cluster (All Nodes) by running hardStop.sh/cmd.
2. install3rdParty.sh -j
         -j = JAR
         -l = Library File
3. Run setupfile.sh
4. Repeat Step 2 and 3 on all Nodes.
5. Restart B2BI Cluster.

Validation Procedure:

If installation was successful then you will see an entry for your new JAR file in dynamicclasspath.cfg file.  

Watch "Sterling B2B Integrator" on YouTube

Sterling B2B Integrator: http://www.youtube.com/playlist?list=PLuG9ugPnSkt5sMigpzbawknri85fFfa52

IBM Sterling B2B Integrator Map Editor Rules Hierarchy

Here is how IBM Sterling B2B Integrator map editor translator controller parse and executes the Rules.

Pre-Session (Variable Declarations & Hard Coded Values)
Root On Begin
   Root Exedend Rules
Record On Begin
Standard Rule & Exedend Rule

Record On End
Root On Begin
Post-Session (Executes before end of map)

Calling B2BI WebService from SOAP UI

soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:mesa="http://www.sterlingcommerce.com/mesa"
   soapenv:Header/
   soapenv:Body
      mesa:DEMO_WEB_SERVICE xmlns:mesa="http://www.sterlingcommerce.com/mesa"
Payload goes here....
      /mesa:DEMO_WEB_SERVICE
   /soapenv:Body

/soapenv:Envelope



Need to add namespace xmlns:mesa="http://www.sterlingcommerce.com/mesa under soapenv:Body.

Updating bashrc file in Ubuntu

Update bashrc file for to default required path variables, xclock setup etc. I find this is clean way then setting up or updating path variables under each user profile unless it is mandatory. This is clean way for sandbox setup at least not for production though.

.bashrc file location:

/etc/bash.bashrc

####Sirish Reddy: Settings ########
PATH=/u01/app/jdk1.6.0_25/bin:$PATH
export LANG=en_US
xhost local:oracle
xhost local:sirishg
xhost local:apache

"Mayor Billy Kenoi, of Hawai'i Island, at HPU's commencement" on YouTube

How To Learn Anything in 20 Hours

Here is the interesting video I came across which teach the logic on how to learn anything in 20 Hours, Interesting... 

Setting up VisualVM from Sterling Integrator

1. Download  VisualVM 

    https://java.net/projects/visualvm/downloads/download/release138/visualvm_138.zip

2. Find the value of the property called ACTIVEMQ_JMX_PORT from sandbox.cfg.

3. Add Remote Host First and under Remote Host add Add JMX Connection. 

    <SI HOST Name>:<ACTIVEMQ_JMX_PORT>

    No need of providing user credentials and Save the connection. 

Enable External Purge in Sterling Integrator

Starting an External Purge

Procedure

1. If you have a cluster installation, make sure that you are not running the external purge on another node of the cluster. If you are running the external purge on more than one node, it will cause database locks. For more information, see Monitoring an External Purge.
   Important: It is the customer's responsibility to keep exactly one of these external purge processes running in the cluster. If you need to shut down the hardware that is running the external purge, you must run the external purge on a different node.
2. Disable the default Purge service. Click Deployment > Schedules, search for PurgeService, and then clear the Enabled checkbox for PurgeService.
   Note: If you have customized the Schedule_PurgeService business process, you need to add your customized code to the new Schedule_PurgeService business process. The new Schedule_PurgeService business process includes a warning message about running the external purge while the scheduled new purge is enabled.
3. Turn off the "Important Schedules are Disabled" email notifications by adding the following line to the customer_overrides.properties file. This removes the Purge Service SQL query for the schedule.
   resourceMonitor.ScheduleMonitor.propertyValue.1=SELECT STATUS, SERVICENAME FROMSCHEDULE WHERE SERVICENAME IN ('BackupService','IndexBusinessProcessService','AssociateBPsToDocs'
   ,'BPRecovery','BPLinkagePurgeService')
4. Do the following:
   • In Windows, do one of the following:
     • Run the command startExternalPurgeWindowsService.cmd from the INSTALL_DIR\bin directory.
     • Run the Windows service. Click Control Panel > Administrative Tools > Services, right-click IBM Sterling B2B Integrator External Purge at BASE_PORT, and click Start.
   • In UNIX, run the command control_extpurge.sh start from the INSTALL_DIR/bin directory.
• In iSeries, run the command
  SBMJOB CMD (QSH CMD('cd INSTALL_DIR/bin; ./ExternalPurgeConsole.sh -clearlocks > INSTALL_DIR/logs/extpurge.log')) JOB(EXTPURGE)

http://www-01.ibm.com/support/knowledgecenter/SS3JSW_5.2.0/com.ibm.help.enhancement_external_purge_524.doc/ArchPrg_StartExtPurge.html?lang=en

Remote FTP Directory Sync using lftp

http://www.cyberciti.biz/faq/lftp-mirror-example/

$lftp user@my.ftp.com
$mirror -c source target

Reverse Sync to Server:
Change local directory to  where you saved the files /home/somedirectory
$lcd /home/somedirectory
$mirror -R

Enterprise Logging with Logstash

Basic Flow of Logstash?

LS Agent(s) (Installed on App Server(s)) --> RabbitMQ --> LS Server --> Elasticsearch Server <-- Kibana <-- end user.

So which scripting language is recommended to parse the logs or data?

There is a bunch of plugins for logstash but mostly we use the grok filter.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

It takes grok expressions which can be the special grok or even ruby regexes with named captures.

If you need help building patterns to match your logs, you will find the http://grokdebug.herokuapp.com and http://grokconstructor.appspot.com/ applications quite useful!

We need to include Puppet in setup or we can manage with out puppet?

Puppet manages LS Agents on all servers. We create filters and apply them based off the server hostgroup and also input files.

Videos on Logstash:

https://www.youtube.com/watch?v=96og3aIgyrc

https://www.youtube.com/watch?v=Kqs7UcCJquM

https://www.youtube.com/watch?v=xj3dUMI263Y

Download Logstash:

https://www.elastic.co/products/logstash

10 Management Tips

Remove SFG Partner Profile (Hard Delete if SFG removal fails)

There was bug in SFG (old versions) where it leaves some orphan entries in DB when you try to remove the trading partner from SFG UI. To solve the problem here are steps to do hard delete of partner profile.

Attempt to delete from SFG in regular way and then remove from Dashboard --> Trading Partner (All Contracts, Document Exchange, Profile, Routing Channels, Mailbox Virtual Roots if any etc.)

Make sure the following table should not return any entry for Trading Partner, If it returns then remove the entry manually...

This example Partner Name is : EDI

select 'YFS_USER' from YFS_USER where UPPER(USERNAME) like '%EDI%'  UNION
select 'YFS_ORGANIZATION' from YFS_ORGANIZATION where UPPER(ORGANIZATION_KEY) like '%EDI%'  UNION
select 'MBX_ACTION'  from MBX_ACTION where UPPER(USER_ID) like '%EDI%'  UNION
select 'MBX_MAILBOX'  from MBX_MAILBOX where UPPER(PATH) like '%EDI%'  UNION
select 'SCI_CONTRACT' from SCI_CONTRACT where UPPER(OBJECT_NAME) like '%EDI%'  UNION
select 'SCI_DOC_EXCHANGE' from SCI_DOC_EXCHANGE where UPPER(OBJECT_NAME) like '%EDI%'  UNION
select 'SCI_ENTITY'  from SCI_ENTITY where UPPER(OBJECT_NAME) like '%EDI%'  UNION
select 'SCI_PACKAGING'  from SCI_PACKAGING where UPPER(OBJECT_NAME) like '%EDI%'  UNION
select 'SCI_PROFILE' from SCI_PROFILE where UPPER(OBJECT_NAME) like '%EDI%'  UNION
select 'SCI_TRANSPORT' from SCI_TRANSPORT where UPPER(OBJECT_NAME) like '%EDI%'  UNION
select 'SCI_CODE_USR_XREF'  from SCI_CODE_USR_XREF where UPPER(USER_ID) = 'EDI%';

select * from YFS_USER where UPPER(USERNAME) = 'EDI';
select * from YFS_ORGANIZATION where UPPER(ORGANIZATION_KEY) = 'EDI';
select * from MBX_ACTION where UPPER(USER_ID) = 'EDI';
select * from MBX_MAILBOX where UPPER(PATH) = 'EDI';
select * from SCI_CONTRACT where UPPER(OBJECT_NAME) ='EDI';
select * from SCI_DOC_EXCHANGE where UPPER(OBJECT_NAME) = 'EDI';
select *  from SCI_ENTITY where UPPER(OBJECT_NAME) = 'EDI'; 
select *  from SCI_PACKAGING where UPPER(OBJECT_NAME) = 'EDI';
select * from SCI_PROFILE where UPPER(OBJECT_NAME) = 'EDI';
select * from SCI_TRANSPORT where UPPER(OBJECT_NAME) = 'EDI';
select * from SCI_CODE_USR_XREF where UPPER(USER_ID) = 'EDI';

--finally delete (be careful) --
DELETE from YFS_USER where UPPER(USERNAME) = 'EDI';
DELETE from YFS_ORGANIZATION where UPPER(ORGANIZATION_KEY) = 'EDI';

Jayaprakash Narayan | Talks at Google

Enterprise Integration Techniques - Notes

Integration Patterns:
   Enterprise Integration Patterns

   Self Managing
 
   Minimize the dependency between components.
 
   John Prosul Robust principle
 
   Be liberal with what you receive and be specific with what you send out.
 
   Always support previous versions (Backward compatibility). It's always hard to support backward compatibility but it's matter of discipline.
 
   Software is craft not a science.
 
   Be wise on Technical Decision vs Business Decision.
 
   InfoQ.com
 
   gotoconferences

   Twitter University

   Emerging Techologies for Enterprise (Conference hosted in Philly)
 
   Agile Stuff
 
   hackernews.com (YCombinator)
 
   HighScalability.com
 
   Netflix Engineering
 
   Thoughtworks Tech Radar
 
   Micro Services (Sam Newman)

Design Patterns Tutorial

Design Patterns Tutorial

Sterling Integrator issue with MSSQL Database

Error while installing - The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "SQL Server returned an incomplete response. The connection has been closed.".
[2014-11-13 17:31:50.499] ERRORDTL [1415899910498]com.microsoft. sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "SQL Server returned an incomplete response. The connection has been closed.".

Solution below link.  -- Alternate Solution:

http://www-01.ibm.com/support/docview.wss?uid=swg21690335

Other useful links.........

http://blogs.msdn.com/b/jdbcteam/archive/2008/09/09/the-driver-could-not-establish-a-secure-connection-to-sql-server-by-using-secure-sockets-layer-ssl-encryption.aspx

https://www.microsoft.com/en-us/download/details.aspx?id=44622

Rain

Harmonium Lessons

http://learnraagabasics.blogspot.com/

Cold storage warehouse with shuttles at -24°C

Encoding a Payload in Sterling Integrator using Java Task

import java.io.InputStream;
import java.lang.StringBuilder;
import com.sterlingcommerce.woodstock.workflow.Document;
import com.sterlingcommerce.woodstock.util.Base64;

Document doc = wfc.getPrimaryDocument();
InputStream in = doc.getInputStream();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < in.available(); i++) {
   sb.append(in.read());
}
byte data[] = sb.toString().getBytes();
byte newdata[] = Base64.encode(data);
doc.setBody(new String( ""+ new String(newdata)+"").getBytes());
return new String(sb.toString());

Sir A.P.J Abdul Kalam Speech

Sir A.P.J Abdul Kalam

Don't take rest after your first victory because if you fail in second, more lips are waiting to say that your first victory was just luck." - A.P.J Abdul Kalam

"All Birds find shelter during a rain. But Eagle avoids rain by flying above the Clouds." - A.P.J Abdul Kalam

"Failure will never overtake me if my definition to succeed is strong enough". - A.P.J Abdul Kalam

"Man needs difficulties in life because they are necessary to enjoy the success." - A.P.J Abdul Kalam

"If you want to shine like a sun. First burn like a sun." - A.P.J Abdul Kalam

"It is very easy to defeat someone, but it is very hard to win someone" - A.P.J Abdul Kalam

"All of us do not have equal talent. But , all of us have an equal opportunity to develop our talents." - A.P.J Abdul Kalam

" Be more dedicated to making solid achievements than in running after swift but synthetic happiness." -A.P.J Abdul Kalam

"Thinking should become your capital asset, no matter whatever ups and downs you come across in your life." - A.P.J Abdul Kalam

" Without your involvement you can't succeed. With your involvement you can't fail. " - A.P.J Abdul Kalam