Other terms include unintentional information disclosure, data leak, data spill, or just plain old data theft.

The Christian Science Monitor has recently compiled the five most expensive data breaches or data theft  as follows:

5. US Veterans Affairs – $25-$30 million – the names, birth dates, and Social Security numbers of 17.5 million military veterans and personnel were stolen in 2006 from a laptop that a Department of Veterans Affairs employee had taken home. The costs to the VA included money for running call centers, sending out mailings, and paying for a year of a credit-monitoring service for victims. The Ponemon Institute, a research organization in Traverse City, Mich., estimates the breach cost at least $25 million.

4. Heartland Payment Systems – $140 million – Heartland Payment Systems, a payment processor based in Princeton, N.J., was the victim of a major cyber attack in 2008.  Criminals installed spying software on the company’s computer network and stole the numbers of as many as 100 million credit and debit cards. Albert Gonzales, a hacker from
Miami, was accused of playing a critical role in the Heartland hack, as well as other massive data breaches of companies including retailer TJX, 7-Eleven, Inc., and the grocery chain Hannaford Bros. Co. Inc. In 2010, Gonzales was sentenced to 20 years in federal prison. The company paid about $140 million in fines and settlements but recovered tens of millions through insurance, Business Insurance reports.

3. TJX – $256 million or more - the Framingham, Mass., retailer that owns national chains including TJ Maxx and Marshalls, estimated that a 2007 data breach would cost the company about $25 million. But in the end, the total cost was at least 10 times as high.

Cyber criminals took more than 45 million credit and debit card numbers, some of which were used later to buy millions of dollars in electronics from Wal-Mart and elsewhere. Mr. Gonzales, who played a major role in the Heartland hack, was linked to this cyber attack as well.

2. and 1. The surprise - who would have thought that the most expensive would be a tie?  I would have bet money that the most expensive would have been a large financial institution, or money changer.  Nope, turns out the tie is between an entertainment company and an advertising company.

Epsilon – to be determined – in March 2011, hackers stole millions of names and e-mail addresses from the Dallas-based marketing firm. Epsilon handles e-mail lists for major retailers
and banks like Best Buy, JPMorgan, TiVo, Walgreen, and Kroger. A study by CyberFactors, a cyber risk analytics company, estimates that the breach could cost between $225 million and $4 billion, depending on what happens with the stolen data, Business Wire reports. Mr. Ponemon offers a lower estimate: at least $100 million, with most of the lost costs going toward losing customers due to a damaged reputation. Ponemon says that because the stolen data was e-mail information, the costs won’t be as high as if financial information had
been stolen.

Sony – to be determined – the Sony data breach, which exposed information from more than 100 million user accounts in April, could prove to be the mostly costly data breach of all time. Hackers obtained personal information, including credit, debit, and bank account numbers in some instances, of PlayStation Network users and Sony Online Entertainment users. After discovering there had been a breach, Sony shut down both networks temporarily. Ponemon estimates that the breach could cost Sony and credit card issuers up to a total of $2 billion.

Relative to their overall revenues and assets, the loses here are big, yet not staggering.  Nobody went out of business.  The thought to ponder is “what would it cost my company were we to lose sensitive information?”  We’ll discuss those loses in another blog.

Top Ten Reasons to Deploy DataExpress in Texas Government

10. DataExpress is on the approved DIR list

• You can “try” before you “buy”
• We can assist you in building evaluation criteria
• No need to go for bids
• Services specific to Managed File Transfer are available

9. MFT is part of the State and National plan

• According to the 2010-2014 State Strategic Plan for Information Resources Management the #1 challenge is:

  • Secure IT Resources
  • Enhance Capabilities of the Shared Infrastructure (Strategy 1.1)
  • Leverage Shared Applications (Strategy 1.2)
  • Leverage the State’s Purchasing Power (Strategy 1.3)
  • DataExpress addresses each of these initiatives

• MeriTalk, a government IT provider, reports in a May 10th 2010 report that when transferring data, Federal Agencies use unsafe methods to move data:

  • 66% moves by physical media (tapes, CDs, DVDs, USB drives, etc.)
  • 60% moves using FTP
  • 52% is transferred via personal e-mail accounts
  • Records lost due to a security breach cost $2.40 per record (Verizon/US Secret Service) (Texas estimates are much higher)

• The Secure Federal File Sharing Act, (March 24, 2009):

  • Prevents Government employees from using peer-to-peer file-sharing software, including FTP

• January 2005 – August 2009:

  • 105 incidents were reported in the public media
  • Involved privacy data from Texas-based organizations
  • 43 of which were government-related
  • universities, cities, counties, and state agencies
  • Exposed more than 3 million individual records
  • Represents more than 12% of the state’s population
  • Cost estimated at $202 per exposed record

8. Free FTP isn’t really free and potentially introduces significant risk

• It’s true every operating system comes with FTP, however:

  • Additional work is required to add security
  • SSL/SSH security is difficult to attain
  • No roles management is included
  • Requires server IDs and passwords
  • Lots of confusion with client/server modes
  • Requires manual configurations outside of the FTP server
  • Audit trails are difficult to understand

7. Removes the need for script writing and developing code

• DIY implementations almost always require script writing
• Requires code for interfacing to other applications
• With DataExpress:

  • All components for transmitting/collecting, packing/unpacking, encrypting/decrypting are part of the product
  • File communications are built-in using common-sense workflow processes that don’t require scripting
  • Features that allow coupling with external applications are available and robust

6. Using DataExpress is much easier than “DIY MFT”, and lets us provide evolving standards

• Designing your own multi-featured MFT solution is challenging
• Important considerations – like security, reliability and auditing – are not adequately addressed
• Time to develop, test and deploy is significant
• Usually single use solutions
• Difficult to embrace changing policies
• Usually ignores DR and high availability
• With DataExpress
• DR and hot-spare capabilities are built-in
• New protocols, packaging, and compression features are added as they become standard
• Scalable as demand increases

5. Works in multiple environments

• Windows Servers
• Linux/Unix
• OS/390 (Z/Linux)
• Solaris
• Minimal reconfiguration required if changing platforms
• Virtual environments supported
• Allows secure DMZ deployments with Secure Gateway option in all environments

4. Everything contained within one framework

• Security-aware and managed within DataExpress
• Robust audit trails are included
• Job-orientated without requiring scripts or code
• Success and failure notification are standard
• Calendars and scheduling are base functions
• Conditional and branch processing functionality
• Browser-based user interface
• Auto-expiration of User IDs
• Auto-expiration of files

3. Unparalleled support in solving problems

• Do-it-yourself solutions require do-it-yourself support
• Due to the products wide distribution, we have probably already solved any problem you are likely to encounter.
• No file transfer protocol is “standard” – we have already developed protocol properties to meet the need to communicate with non-standard clients
• 7 x 24 x 365 support available
• Wiki’s and blogs available
• Online problem reporting and tracking

2. Security, management and auditing already built-in

• Roles-based user and operator management
• Robust audit trails showing all communication events
• All communication events are recorded in a database enabling reporting to summarization and detailed levels
• Near-real-time monitoring
• Relationship among jobs, files, people, and locations are enforced through roles-based relationships

1. Proven solution, works already, provides peace of mind

• DataExpress successfully delivers and collects hundreds of thousands of files daily for our customers
• Some of the largest financial institutions in the world use DataExpress to ensure successful Secure MFT
• These files represent billions of dollars; and have significant penalties if SLA terms aren’t met
• DataExpress is completely scalable to any size organization
• Built-in active recovery and planned DR deployments
• 7 by 24 by 365 expert support available

•  50% general infrastructure knowledge: server Operating Systems administration, networks, file management, internal policy requirements, script writing, and firewall negotiation
• 45% managing relationships and procedures with clients and internal business units, managing file transfer processes, service level management, set up, minor problem resolution, audit reconciliation, solving communication issues with remote sites
• 5% managing DataExpress, severity 1 issues, user roles, software installation (updates, initial configuration)

 In a non‐managed file transfer environment, all of the tasks performed can be considered to be in the general infrastructure knowledge category. This usually requires one or more highly skilled system administrators to organize and manage the file transfer environment. This mode of operation usually results in custom scripts, poorly documented processes and procedures, limited scalability, and nearly no continuity, all translating to potential risk and end user frustration.

Also, time and time again, we have seen system administrators with hard earned, expensive institutional knowledge leave or change jobs. More risk and more frustration. Moving to a centrally managed file transfer system alleviates issues that arise when system administrators with hard earned, expensive institutional knowledge leave or change jobs. Implementing DataExpress technology provides a standardized solution where policy and procedure (implying compliance to mandates,) are documented, implemented, and audited within a defined area. This is far superior to attempting to enforce policy distributed to a number of discrete (usually undocumented and not inventoried) throughout the infrastructure. Often these types of implementation lead to missed compliance and risk – while at the same time are far more expensive to build and maintain.

 If your experience is different from our understanding, we’d like to hear about it.

This article is a little out of scope for my normal meanderings, but it is important.

Long predicted, it has now happened that the internet world has used all the addresses that are available under the IPv4 format.  But that’s not the end of the world, you’ve already got your block reserved or used, right?  Still not the end of the world, most likely you already have your address spaces already clearly advertised and known to your trading partners and end-points.

So if you haven’t already done so, you should make a plan for moving to IPv6.

Good news from our side is that DataExpress is already compatible for IPv6.  For DataExpress for NonStop, Release 7.6 and for DataExpress Open Platform, release 4.1 are tested and known to work .

Follows below is an article from AFP that does a really good job of explaining IPv4 versus IPv6.

See: http://news.yahoo.com/s/afp/20110203/tc_afp/usitinternetsoftwareicann.

MIAMI (AFP) – The global warehouse for Internet addresses ran empty on Thursday.

The non-profit Internet Corporation for Assigned Names and Numbers (ICANN) doled out its last five batches of “IP” numbers that identify destinations for digital traffic.

“A pool of more than four billion Internet addresses has been emptied this morning,” ICANN chief Rod Beckstrom said at a Miami press conference.

“It is completely depleted. There are no more.”

He brushed aside fears of modern life being devastated by an “IPocalypse,” saying Regional Internet Registries (RIRs) worldwide will be doling out remaining addresses to support a shift to a bountiful new “IPv6″ format.

“It is like running out of license plates,” said Internet Architecture Board chairman Olaf Kolkman. “Driving on the road the next day would not change.”

The touted solution to the problem is a switch to an “IPv6″ format which allows trillions of Internet addresses, while the current IPv4 standard provides a meager four billion or so.

The effort and expense of changing to IPv6 would fall mostly on Internet service providers, websites and network operators that have to make sure systems can handle the new online addresses and properly route traffic.

“If an ISP (internet service provider) gets its act together, it shouldn’t be a massive problem,” Trefor Davies, chief operating officer of British ISP Timico, told AFP.

“We really should see this as an historic event,” he continued. “The very nature of the Internet has changed with the transition.”

Beckstrom expected the full switch to IPv6 to take years with potential overall costs in the billions of dollars, some of which could be factored into routine replacement of equipment.

“We are talking about billions of dollars here globally, not trillions of dollars,” Beckstrom said.

Consumers, for the most part, should remain oblivious to the switch since complex IP numbers would still appear to them as words and domains, such as icann.org.

“My mother, my neighbor, my kids — they should never notice,” Kolkman said.

Some people might need to update routers or modems that connect computers to the Internet.

“All conditions are in place for a successful IPv6 transition,” Beckstrom said. “The future of the Internet and the innovation it fosters lies within IPv6.”

ICANN has been calling for a change to IPv6 for years but websites and Internet service providers have been clinging to the old standard since the birth of the Internet.

With about seven billion people on the planet, the IPv4 protocol doesn’t allow for everyone to have a gadget with its own online address.

The situation has been equated to not having enough telephone numbers for everyone.

The number of addresses that IPv6 allows for amounts to 340 “undecillion” (followed by 36 zeroes); enough for a trillion people to each be assigned trillions of IP numbers, according to Beckstrom.

IPv4 addresses were expected to run out first in Asia, where demand has been highest as people and businesses in emerging markets embrace online lifestyles.

Once RIRs run out of IPv4 addresses, they will turn to IPv6.

The formats have been likened to different languages, with translation needed for systems to handle both.

Computers and other gadgets that don’t get the new format might have to start sharing instead of having unique identifying numbers.

“The Internet won’t stop working; it will just slowly degrade,” Google engineer Lorenzo Colitti said of not making the move to IPv6. “Things will get slower and flakier.”

Google, Facebook and other major Internet players will add IPv6 addresses to their systems in a one-day trial run on June 8 to let all parties involved check for trouble spots.

“We need to kick the tires on it at a global scale and see if there are some unforeseen problems,” Colitti said. “There is really a rallying cry element to it. No single player can do it alone; we need to work together.”

World IPv6 Day will start at 0001 GMT on June 8.

Adoption of IPv6 is vital to preventing the Internet from becoming “balkanized” with localized addressing frameworks, according to Internet Society chief technology officer Leslie Daigle.

ISPs and networks worldwide have implemented IPv6 or plan to do so, Daigle said.

“It’s only a matter of time before the RIRs and Internet Service Providers must start denying requests for IPv4 address space,” said Raul Echeberría, chairman of the Number Resource Organization, an RIR umbrella group.

“Deploying IPv6 is now a requirement, not an option.”

Unless you are still doing business on a fax machine or sending reports or data by mail, you most likely have running somewhere within your organization at least one file transfer application.  And the chances are that at least one of these applications is critical – or at the very least important – to the operation of your enterprise.

Historically file transfer is taken for granted within many organizations.  Most started with a single purpose need, then later expanded to include other processes, some related, some not.  What concludes is a disjointed effort that is not managed collectively.  Departmental projects, multiple support organizations, a limited view to information management are many of the sources.  Electronic file transfer used to be a competitive advantage, now it is a competitive necessity.

Your most prized asset is your product; the second most prized asset is information about your product. 

When considering strategies for MFT, consider the following:

• Tactical versus strategic
  • Why even consider a tactical solution when strategic builds a blueprint –
  •  the work effort is about the same, or less for strategic
• No product is PCI/ Sarbanes-Oxley/HIPAA,-compatible, MFT only a cog in the whole gears – where “torque” is the delivery
• Balancing among risk, business need, obstruction, competition, and ignorance as to importance to the enterprise

DataExpress is an effective method of consolidating and managing all your file transfer requirements under a single unifying management concept.  It allows for an infrastructure where you can migrate your existing systems to DataExpress without having to change everything all at once.

Although we use “managed” when describing DataExpress, a more appropriate descriptor for DataExpress might be “orchestration.”

Orchestration does not necessarily mandate that all file transfers traverse a single server, it means that all file transfer activities are governed by the same overarching set of policies and procedures.  Orchestration has the following attributes:

• Defining policy – all follow the same rules and aggregate the details of end-point and business centric relationships, including activities  
• Uses the same reporting mechanisms and criteria
• Uses same metrics for measurement across all platforms
• Continued auditing, monitoring and reporting
• Assurance of uniform enforcement – for mandated controls and standards
• Problem remediation and continuous improvement
• Forecasting
• SLA measurement
• Risk identification, avoidance, and management

So getting back to our definition of Managed File Transfer. Far too often we speak with potential customers whose idea of managed file transfer is that “we finally managed to make our own solution work.”

Wikipedia describes file transfer as a generic term that applies to the act of transmitting files over a computer network. I would suggest that transmitting data as a controlled streamed flow of bits over a network is a more apt description. The concept of a file is simply a logical container of data which has a logical beginning and a logical end.

Managed is a vague terms that can mean a lot of different things. Managed File Transfer usually refers to a software solution designed to handle file transfer. Most general purpose operating systems have built-in facilities for effecting file transfer. OS-based file transfer facilities are generally not robust enough to provide a truly adaquate or secure method of managing file transfer.

Emphasis should be placed more on the management aspect of data transfer and less on the physical act of moving data bits from one location to another. Often we spend more time on the discussion of how rather than the why of moving data. I should stress that being in control of data movement and having governing enforceable policies is of more importance than the actual program or system used to move data.

Inside the industry we use the acronym MFT to encompass all the attributes that surround file transfer techniques. Wrappers – logical files – is the way in which we identify select units of work effort. Included in MFT are other features such as scheduling, protocol selection, automation, reporting, auditing, policy identification and enforcement, performance, capability, error trapping, and reporting.

Here are some defining attributes of file transfer:

• data logistics – tantamount to an electronic supply chain of information
• system-to-system, partner-to-partner, and person-to-person

Organizational

• Application-to-application, internal/external
• File-centric Business Process Relationship

Other features of MFT are generally included in the implementation of a managed system. These features may be stand-alone, or incorporated as part of the overall process. Each will be covered in detail in later postings.

Encryption – ability to encode data in such a way that only the sending and receiving parties can view it. Encryption takes place in two distinct areas, data itself before and/or after transmission; and encryption of the data transfer session through protocol. These terms are not mutually exclusive.

• Auditing – ability to track every activity associated with file transfer, such as who sent a file, when they sent it, who received it, when it was received, and so on.
• Non‐repudiation – ability of a file transfer system to ensure and prove that a file was received by the correct recipient.
• Compression – ability to reduce the number of bits required to contain a file. Compressibility is always dependant on data content and structure.
• Automation – ability to pre-schedule and predict file transfer events without manual triggering.
• Error trapping and reporting – ability to capture and handle unplanned events that interrupt the timely flow of data.
• Contingency planning – forecasting, pre-planning, and provisioning for backup/failover and disaster recovery.
• Data examination – ability to take prescribed action depending on file contents and not file name.
• Data reformatting – ability to reorganize, summarize, restructure data contained within a file before or after transmission.
• Protocol conversion – ability of a system to accept input from one protocol and write output to another protocol during data streaming of a logical file.
• Proxy services – ability or facility for a client to request services of a server on its behalf. File transfer proxies are used to isolate and protect internal clients from hazards of public-facing presence.
• Digital signature – a mathematical scheme for demonstrating the authenticity of a digital message or document.
• Certificate Management – storage and management of digital certificates and keys used for validation, identification, and encryption.
• Ad hoc – ability to effect file transfer sequences manually on demand.
• Mailboxing – ability to create public or group-centric repositories for sending/receiving files.
• Data archiving – ability to expire and/or offload processed files to auxiliary storage.
• Application interoperability – hooks, interfaces, triggering mechanisms to applications and processes outside the file transfer system.
• Clustering/load sharing – ability of a file transfer system to manage workloads among multiple platforms while retaining central management and control.
• Provisioning – ability to modify or create new data transfers without requiring manual scripting or system-level users. Also provides inventory of partner information for management purposes. Allows for file transfer protocol changes without dramatic editing.
• Policy and procedure driven – allows for managing to an overarching set of corporate policies that determine security and data management guidelines. Consolidates all file transfer activities into a well-defined set of rules and auditability. Allows for incorporation with change management control within the organization.
• In-flight and historical reporting and trending – ability to examine data transfer performance for currently actively data transfer sessions as well as reporting historical performance.
• Roles-based authority – ability to isolate files, groups of files, and operator/user rights and capabilities bases on hierarchy of rules defined for users and systems.
• Multi-level authentication – US Federal regulators consistently recognize three authentication factors:

1. Something the user knows (e.g., password, PIN);

2. Something the user has (e.g., ATM card, smart card); and

3. Something the user is (e.g., biometric characteristic, such as a fingerprint).

Authentication methods that depend on more than one factor are more difficult to compromise than single-factor methods.

Go to http://en.wikipedia.org/wiki/Telex to find the history of telex.  For a brief history of teletype see: http://en.wikipedia.org/wiki/ASR-33_Teletype.

Somewhere along the same time line for teletype and TWX development came the rise of IBM and the world of the mainframe computer.  With that growth and expansion, came the need to submit jobs and route print output to/from the central IBM mainframe to remote locations.  The machinery at the remote site gained the term Remote Job Entry, or RJE.  Used primarily for input and output to the mainframe, someone along the way came up with the idea to use this technology to move actual data from mainframe to mainframe.  That was a great idea, except for the added step of breaking/concatenating files into record lengths  as even multiples of 80 byes, the data content capacity of a punched card or Unit Record Device.  For SNA, other record lengths were supported but only to control equipment operation over the different communication protocol.

Generally speaking, the IBM communication protocol Bisync is tantamount to moving EBCDIC data – although in reality it doesn’t have to be limited to EBCDIC code sets.  Bisync is the communication protocol and not the code set definition.

Bisync file transfer is still alive and kicking in the business world.  Its demise has been predicted for years, but it still hangs on.  Later I’ll talk about inventorying your business partners and how to pursue moving away from this protocol.

In the rest of the world – non-EBCDIC or ASCII – machines used a different character code set.  For machine-to- machine communications in this world, records were terminated by an extra character. 

Well consider the problem of communicating form an ASCII-based machine to an EBCDIC-based machine. ASCII code sets generally use hex “0D”/”0A” – CR/LF (carriage return/line feed) as the default record terminator. (See telex/teletype RJE devices above.)  After all if you’re sending print to an ASCII printer, you need to tell the machine to return the print carriage and advance the paper one line – not at all unlike a manual typewriter.

EBCDIC-based devices didn’t see the need for a CR hex “0D” as a communication control character to indicate a request for retransmit that block of data.

Ok, so you are confused as to why this is important. 

If you are exchanging files where the computer types are the same, you don’t really care.  But if you are tasked with moving files among UNIX-based platforms and Windows-based platforms, then the problem raises its head because UNIX-based platforms use only hex “0A” characters to signal end-of-record.  Don’t confuse this with the code set that represent data.

I see I haven’t yet addressed the question – What is managed file transfer?  My intent in this article is to show that file transfer has always been around in one form or another.  And since its very beginning there were/are unanticipated problems to be solved. 

I hope to help you understand how carefully choosing your strategic file transfer solution should leverage the experience and legacy knowledge of a reputable vendor.  Handling issues such as I’ve described above should be a non-issue if the product selected is robust enough.  If you are creating your own solutions, what I’ve described is just one of many issues you’ll need to address.

It’s 2011 and here at DMBGroup we’ve been making plans to pass along our knowledge and considerable experience on file transfer systems within the business and corporate communities.  We considered publishing a huge whitepaper in hopes of capturing the universe of information about this subject.  But then, who would read it?  Doesn’t most everyone already know that file transfer is “free” and “nobody really has a canned solution that really meets our needs” and so forth.  So what would yet another whitepaper do for you – or for us for that matter?

We’ve decided to present assorted ideas and discussions around many components of file transfer. Our goal with this blog is to inform, not instruct, and point out the values DataExpress offers. We hope that it will bring insight into recognizing and managing your file transfer environment – whether you use DataExpress or not.  Ultimately we hope we can show how DataExpress can solve some of your problems.

Via this communication tool we will publish articles on many of the various components necessary for managed file transfer.  We will cover at least the following topics:

• What is Managed File Transfer?
• Why Have Managed File Transfer?
• What types of MFT are there?
• Implementing and Managing file transfer?
• Inventory and provisioning
• Security
• When is enough, enough
• Compliance
• Auditing

We invite your input, comments, and requests for more topics, or delving deeper into a particular area.  Better yet, we invite you to ask us for more information about DataExpress and how it can help you.

Again, welcome to our blog about Managed File Transfer.

Wilson B. (Sonny) Snipes

sonny.snipes@dmbgroup.com