|Contents||Bulletin||Scripting in shell and Perl||Network troubleshooting||History||Humor|
|News||Missing backup horror stories||Partimage|
|Emergency Restore Procedure||Baseliners||Software RAID||Ext3 filesystem||LVM||Linux backup and recovery||Shadow IT|
|Perl Backup Scripts and Systems||Linux Swap filesystem||Booting into Rescue Mode||Grub||How To Manage Your Disk By UUID||Data deduplication||Linux filesystems|
|Linux Multipath||udev||Labeling the partition||Loopback filesystem||Sysadmin Horror Stories||Humor||Etc|
There are two typical scenarios for disasters in datacenter, which often result in loss of data:
The key question after most computer disasters, especially those that require bare metal recovery is: do your have a good backup, So the first task is to locate and check them all. Typically you have two types of backup:
The key tool after the disaster is your cool head: the most damaging things are usually done in a hurry, without thinking out possible consequences.
The second consideration is simplicity: disaster recovery procedure typically works if and only if it is simple, well understood by personnel, and was tried many times before. In this sense typical enterprise backup tools such as HP Data Protector can create a horrible mess: it tend to stop making backup on its own and personnel often overlook this fact until it's too late. With the capacity of USB drives sysadmin can make "private" backups -- which represents positive example of "Shadow IT"
If you are lucky you have separate user data backups for which usually are done using enterprise backup. It may (or may not) exist or be current (Data Protector sometimes stops making backup for no reason, so even with enterprise software there is no guarantee), but also a recoverable OS backup (rescue image).
And good means recent (in most case the preferable is from yesterday) and readable. In this sense having a local backup OS on USB stick such as Cruiser Fit (which is now up to 128 GB) makes perfect sense as amount of data in "pure" OS backup is usually minuscule (around 8GB for most versions of Linux)
If the hardware has been damaged, the parts should be replaced before attempting to use the computer.
In Linux you usually recover each filesystem separately. First you need to repartition the drive. The easiest way to do it in RHEL is to use Kickstart file, which contains partitioning instructions you use in initial OS install. After that you ca use tarball, cpio or other backup tool to restore partition by partition.
Re-mount the root partition of the read-write (mount -u -o rw /mnt). Use your backup program and backup tapes to recover the data for this file system (e.g. restore vrf /dev/sa0). Unmount the file system (e.g. umount /mnt). Repeat for each file system that was damaged.
Once your system is running again, backup your data on a regular basis. Whatever caused the crash or data loss may strike again. One hour spent on backup today, may save you days of efforts to recover the data later. In a sense, anydisaster recovery strategy is a bit like home insurance; you hope that you’d never have to use it And here like everywhere else prevention is better, much better than cure.
Minor discrepancies, omissions and oversights in an organization's disaster recovery plan and tools can have a major impact on the time required to recover from a disaster and the associated business impact. Please note that recovery of data directly from a damaged harddrive costs many thousand dollars. In many cases it is possible, but it si very costly.
While most organizations claim to have some form of IT disaster recovery plan in place, there are wide-ranging differences in quality, scope and detail level from one plan to another. It’s best to avoid cutting corners in this area.
Aug 01, 1994| Linux Journal
It's happened to me more than once. It will probably happen to you at one time or another. You turn on your PC, expecting yet another fun session of pure Unix power, when something goes wrong. It won't boot; hard drive not found; it just hangs. Now what? This article will help you figure out what is wrong and get started with fixing it. Read this article before something goes wrong, and it will be
Something has gone wrong. That's all you know. Staring at your blank or garbage-ridden screen, the only thing you can think is “Now what do I do?” Even if you have not had this happen yet, there is probably a good chance you will face this. With all of Linux's power, it is still rather easy for a new—or even experienced—user to make a mistake and mess up something.
With some advance preparation, this kind of situation won't leave you stranded. Make sure you know how to track down a problem, have a bootable disk, and have a set of rescue disks, configured for your particular setup.
Your first step is tracking down the problem. Do you get to the `Uncompressing Linux...' message? If not, your problem is with the boot disk or LILO. Having a spare boot disk should allow you to boot your system, and then you can reconfigure LILO or make a new boot disk.
While Linux is booting, do you get past the partition check? If so, your hard drives are probably fine with Linux. I had a hard drive once that made Linux hang when it tried to find the partitions. The drive didn't work in any other system I tested, so the drive was bad.
Also, if you get past the partition check, then the kernel is not your problem. After the partition checks are done, root is mounted and then /etc/inittab is read. As you may or may not recall, /etc/inittab is used by the init program to start login processes and begins reading your /etc/rc files to mount your partitions, start your network among other things. Once the inittab is read, it goes to the corresponding file for mounting additional filesystems, starting network services, and other startup services. If you see your filesystems being mounted, that means that some of your rc files are being started.
Once the inittab is read, it goes to the corresponding startup file (“rc file”) for mounting additional filesystems, starting network services, and other startup services. If you see your filesystems being mounted, that means that some of your rc files are being started.
Finally, make sure that your network services are starting if you want them started on your system. This is one of the final parts to the startup sequence.
Now, what do you do if you know you have a problem? Before you get into a jam, make sure you have backups. If things get too bad you can always re-initialize your partition and restore from an old backup. Also make sure to have backups handy of your /etc directory.
One good idea is to get a copy of the rescue disks available through FTP. These disks will allow you to boot linux from a pair of floppies and access most of your partitions. This way, even if you can't boot because of a bad /etc/inittab file, you can still boot linux and get access to the bad file, then fix it.
Some of these rescue disks come completely ready-made, so that you can use the rescue disks very easily. The disadvantage to these sets is that they may use an older kernel, may not have some pieces that you need (SCSI support, for example), and may not have the set of programs that you want to see in a rescue disk.
There are other sets of rescue disks where you specify which programs you want to include. They also use the current version of the kernel that you are using. The drawbacks to these are that you need to know what you are doing and they take a bit more work than simply getting a pre-built rescue disk. Two such packages are SAR (Search and Rescue) and rescue. Each of these packages is small, as they both use programs that are already on your system.
If you have two floppy drives, you can go through the rescue disk(s) and find out what programs that you'd like to add, such as your favorite editor. Usually one disk can contain all the programs you'd need in the event of a disaster, but having two disks chock full of utilities will be even better. Here's how:
First, put a floppy in your second drive. I have a 5.25 HD drive as my second floppy, so I'll use that in my examples.
The fdformat program is used to low-level format a floppy. Its syntax is:fdformat <device>
where <device> is the name and type of drive you're using. For example, I have a high density 5.25" drive as drive 2, so my <device> would be /dev/fd1h1200. A high density 3.25" would be /dev/fd1H1440.
Now you put a filesystem on it. Use the same filesystem that you are using on the root partition of your system. In my case, that would be the Second Extended Filesystem (ext2). So, let's put a filesystem on my floppy:mke2fs -c /dev/fd1h1200
Replace the /dev/hd1h1200 with /dev/fd1H1440 if your second drive is a 3.5" high density drive.
Now you should have a filesystem on a disk. Mount it on an unused directory. The /mnt directory is usually used for this. If /mnt does not exist on your system, domkdir /mnt then do mount -t ext2 /dev/fd1 /mnt
Your disk will now be mounted on /mnt. At this point, start copying over whatever programs you want. Make sure of two things:
- Make sure that the shared libraries on the rescue disk will work with the programs that you put on the disk.
- Make sure that you copy over all the files you need. Some editors have configuration files or help files you may need.
If you are using a rescue disk such as SAR or rescue, you won't need to worry about libraries and you can skip ahead a few paragraphs. Or you can read it and get a better hint about how the shared libraries work.
The idea behind shared libraries is that many common C functions get included in one file in a common location. This saves a lot of space as those common functions no longer need to be duplicated in each program binary. The drawback is that it is a tiny bit slower because now two files have to be loaded instead of one. For the toss-up between speed and size, I'll take the size, especially on a floppy with very limited space.
Another small problem with shared libraries is that programs compiled to use a new library won't work if the only library that is available is an older one. For example, a program compiled to use version 4.4 of the libraries won't work if the only set of libraries available is version 4.3. You'll wind up getting an error message about incompatible libraries. If this happens, get a new copy of the libraries or recompile the program to use an older library.
[Ed. Note: this is not strictly true. With modern libraries, the user will get a message, but the program will still try to run if all the necessary symbols are there. For instance, I'm running some binaries compiled under libc 4.5.8 which run fine with my libc 4.4.4, other than giving an error message. I don't know if you want to deal with this or not; probably not.]
To check what versions of libraries the programs are looking for, use the ldd command:ldd <program>
This will return the version of libraries that the program was compiled under. ldd /bin/write for me returns:libc.so.4 (DLL Jump 4.4pl1)
If the files in the /lib directory are libc.so.4.4.1 or above, it will be fine to put the `write' command on your disk. If the library needed is newer than the library on the rescue disk, then you would need to find an older version of the program and put that on the floppy. For example, if the library on the rescue disk was libc.so.4.3.1, I'd need to find an older version of write to put on the disk, or else put libc.so.4.4.1 on the disk.
You don't need to put just executables on this disk. A copy of gzip and a bunch of HOWTO files can come in quite handy as well. Here's a list of suggested files, all available through FTP or on many BBSs. Some of these files may be on the rescue disk you have. Make sure.
Take any of these editors. I find that ed is small and compact, but not much fun with heavy editing or large files. For you, joe may be worth the extra 98k it takes up. If you are unfamiliar with joe or ed, you can use vi, which is a standard program on just about all UNIX systems:joe editor 133kvi editor 101ked editor 35k
General Everyday Utilities:diff 61k (finds changes in big files)grep 61kgzip 46klilo 40kMAKEDEV 9kmknod 3k
Backup utilities:This will vary depending on how you did your backup. You may want a copy of tar, afio and ftape. Get some utilities for the filesystems you run:e2fsck 35kmke2fs 20k
Get some HOWTO files (compress with gzip for real space savings!):Installation-HOWTO 48kSCSI-HOWTO 41kFtape-HOWTO 18k
One more thing you'll want on-hand is a list of all of the cards that are in your machine, the IRQs that they use, and whether they are used by Linux or not. Sometimes a problem can be an incorrectly configured kernel or card.
If you keep these disks set aside and updated often, you'll be ready for anything that might happen.
Tip of the month: When you hit the backspace, do you see /'s followed by the character you just backspaced over? Don't you hate it, too? It reminds me of reading The Unix Programming Environment. Get a new copy of agetty and this should cure the problem. A copy distributed with some Slackware releases had this problem.
A dreadful time has come, a Linux system no longer boots up properly. Whether this was caused through our own curiosity, hardware failure, a friend or co-worker "helping out" is the past. This article will you bring back precious data or functionality of a Linux machine. We can no longer access valuable data and services that our hitherto always available Linux system used to provide. Our intended audience is fledgling administrators, but we might have a trick or two for the old school guru as well.
The first task is simple: determine what you want to bring back. Generally this means restoring the ability to get to a shell prompt, all file systems mounted and network connectivity restored. This functionality is usually provided by runlevel 3. For more information on runlevels, read man page for init, section 8 (shorthand is man:init(8)). The more specific the recovery goal, the simpler our planning. We'll walk through diagnosing the problem, some specific fixes, and refer to documentation for additional information.
Recovering any system to full and proper functioning is not required in many instances. If the jewel to be recovered is data in a directory or on a partition, booting to another operating system (on floppy disk or hard drive) and copying the data may be all that is required for minimal system recovery. System recovery may be recovering data, normal booting, network access, a specific application or user data. If your user data (/home) is on a separate partition, reinstalling Linux and losing the system configuration may be the fastest method of restoring user data. Since Linux has more advanced techniques for system recovery beyond reinstalling the operating system, we'll walk through common disaster recovery techniques. Following are some of the more common disasters and how we recovered from them.Determining the Problem and Possible Fixes
Now that we've decided what to recover, in this case booting into mutli-user mode, we need to determine what the problem is. Our initial troubleshooting practices should ascertain where the problem is: hardware or software, configuration or libraries, etc.
Narrowing down the source of the problem is done in stages, as each stage of booting Linux is completed. Actual fixes depend entirely on your specific environment distribution and problem. Please read the noted documentation and understand the techniques before applying them blindly.
Stage 1: LILO (Linux Loader) Stage 2: Loading the kernel Stage 3: Mounting the Disks Stage 4: Startup Scripts Stage 5: Runlevel Scripts Stage 6: Providing a Login Prompt
After the BIOS screen, LILO should run. If LILO runs, we can safely assume that the most of the hardware is functioning, and that the Master Boot Record or MBR, is still loading. Each of the letters in LILO signifies a different part of LILO has loaded. The most common problem is seeing "LI" and then nothing else. A common reason is another operating system like NT loaded on top of Linux. To resolve this problem you need to wipe out the broken MBR (using Windows "fdisk /mbr" or "dd if=/dev/zero of=/dev/hda bs=512 count=1" and reinstall LILO (/sbin/lilo) back into the MBR. A current boot disk is needed to run /sbin/lilo or dd.
Other common problems for LILO include: errors in /etc/lilo.conf, limitations in the BIOS pointing to IDE drives, installing Linux to a FAT partition and running defrag within DOS, etc. Most problems can be fixed by verifying the options and mappings inside /etc/lilo.conf, running /sbin/lilo and rebooting. I always create a /boot directory with the kernel, bootstrap (boot.b) and system.map near the beginning of my disk to avoid cylinder (1024cyl) and size (>2GB) issues with the BIOS. When the kernel is located on a SCSI drive, put the key word linear in the lilo.conf. Basic documentation for LILO is provided in man:lilo(8), man:liloconfig(8) and man:lilo.conf(5) and extensive documentation for LILO is in /usr/share/doc/lilo/manual.gz.
Loading the KernelAfter LILO has run, it hands off control to the kernel image listed in the lilo.conf. If the kernel image is corrupted, messages vary from silence to core dumps. If someone recently upgraded the kernel, they should have made a boot disk with functional kernel, and kept the old kernel bootable under a different alias. Use the boot disk or boot to the older kernel and work on creating a new functioning kernel. If the original kernel is corrupted, you may find a copy of your kernel on the distribution cd or inside "/usr/src/linux"
Problems loading hardware might also be listed during this time; the log is located in /var/log/dmesg. Additional or alternate kernel log locations are boot.log or kern.log depending on distribution. HOWTO documents and distribution specific kernel guides are your best references for creating kernels. Problems with a particular section of the kernel (advanced power management for example) are most directly addressed by the kernel mailing list. www.kernel.org is the Linux kernel homepage, with additional references and documentation available through web site links.
Mounting the Disks
The kernel loads, then mounts the partitions listed under /etc/fstab. man:fstab(5) provides file format and directive information. All of the mount points must be accessible during boot time. If a mount point fails, the system will prompt for root password (configurable and varies across distributions) and then boot to single user mode (Run Level 1). If a disk volume is not accessible during boot, it's time to use single user mode and comment out the mount line in the /etc/fstab for a temporary fix.
If a disk was not cleanly unmounted, fsck will check for errors with automatic settings. These settings will resolve simple problems but actual errors need to be fixed by running fsck manually in Single User mode. Fsck should only be run on unmounted filesystems. Read man:fsck(8) and man:e2fsck(8) for more information.
If the superblock has been corrupted, fsck can restore one of the backup superblocks. The location of the backup superblock is dependent on the filesystem's blocksize. For ext2 filesystems with 1k blocksizes, a backup superblock can be found at block 8193; for filesystems with 2k blocksizes, at block 16384; and for 4k blocksizes, at block 32768. To use one of these superblocks, run fsck -b [blocknumber] /dev/[harddrive]. For example, "fsck -b 8193 /dev/hda" for the first IDE drive with 1k blocksizes. ,b>fsck -B [blocksize] doesn't require any math, only knowing the block size used.
When using partitioning utilities from several different operating systems, an inconsistency may develop regarding the boundaries of the partitions. The starting and ending blocks will be listed twice when fdisk displays the partition table. This is rare, but does happen on multiboot systems. My recommendation is to always use a 3rd party utility that consistently understands all the filesystems used, specific to all the versions of all the filesystems (NT with SP3, SP4 and SP5 and Windows 2000 all have different versions of NTFS for example) or use utilities like fdisk, cfdisk, sfdisk, disk druid, etc under one operating system. Spoon feeding an operating system a blank partition, of a certain type or a formatted filesystem generally works, some OSes require blank space. Last time I had this issue, I used a popular third party partitioning software and it resolved the issue.
Startup ScriptsStartup scripts are very distribution specific. Most distributions place the scripts in /etc/ or /etc/rc.d/ with the startup scripts in either rc.boot or rcS.d. In addition to the actual mounting, hardware detection occurs, networking is configured, hostname specified, clocks started, portmaps rendered and console settings declared. This is happy stuff that we need for the next section. If one of these scripts fails, find out what hardware it depends on (isapnp for sound card failed? then maybe you have a sound card hardware issue or a kernel module issue). If any complaints or error messages are displayed, few problems will stop the boot sequence. Research the script in use and look for man pages for these scripts. Try man -k "scriptname" if you can't find documentation.
Runlevel ScriptsAfter executing the initial startup scripts, either runlevel 3 (multiuser shell) or 5 (Xwindow) will be invoked depending on the configuration in inittab. Read man:inittab(5) for differences between run levels and initial configurations. PCMCIA is started along with the user, system and network daemons. Any number of these can fail, depending on configuration. Both Apache and Sendmail will hang without a proper hostname, etc. The daemon failing should be reported on the screen and in the normal messages log. Check the software documentation or turn off the daemon in question. chkconfig --list will display run levels and daemons under Redhat and mandrake. Linuxconf has a module for controlling the behavior of different services. Debian start up scripts should be managed using update-rc.d in accordance with the debian policy manual.
The login prompt relies on a subsystem and a set of programs, in addition to the username and password validation. If there is a problem with logging in, common problems include caps lock, forgotten passwords (recover root password from single user mode), trojan login scripts and rootkits. Most trojans and rootkits try to convince users that the system was not comprimised by providing a moderately consistent user experience. Validate packages using rpm -V or files using a file verification program like tripwire if "strange" behavior is noticed. www.securityfocus.com lists extensive security related resources, and provides information about rootkits, trojans and utilities like tripwireRecovery Tools of the Trade
Single User Mode
Single User Mode is extremely useful for working on a sick system. Minimal system configuration is loaded, most file systems aren't mounted, and the Bourne Shell, sh, is loaded without any profile and basic environment. Single user mode can be accessed several ways. While selecting a image in LILO, single user mode is one the options that can be specified, e.g. "Linux single" Additionally, single user mode can be set from multiuser mode "telinit 1" and opportunities to access single user mode are offered when critical services fail.
Making a boot disk can be done with utilities or by copying the boot.img file from the Redhat CD. Redhat and Mandrake offer mkbootdisk for creating up to date bootable floppies. Create a bootdisk with the current kernel with the command "mkbootdisk `uname -r`" Debian offers mkrboot which creates kernel and rootimage bootable from a single floppy. Most distributions also offer bootable floppy images that need to be copied from the distribution cd to a floppy disk. The dd command "dd if=[path]/boot.img of=/dev/fd0" will copy the boot image to floppy, bit by bit. This disk will be bootable, but will not contain a current kernel or drive map.
With versions of Redhat prior to 6.2, a rescue image was also provided, containing basic file utilities to restore the system. Text editors and disk utilities provide a basic level of support to restore the system to a bootable state.
Some Linux partition utilities are gpart and parted which offer partition tools and recovery. I have used the commercial product, Partition Magic, successfully in the past to correct many filesystem and partition problems.
When all else fails...
After the utilities and recovery modes have failed, or the problem is too complex to resolve through simple means (like the time I deleted over half of the software packages on my system including all the text editors) then reloading the operating system will provide the most direct means of recovering the system. User data may be preserved, but system configurations will be lost. Setup varies according to distribution and configuration. Redhat's "Server" installation wipes all partitions, "Workstation" installation preserves some partitions and installs into blank space. "Custom" grants the most control and should be used for reinstalls. Debian installation is user controlled and very flexible, and partitions are mounted and assigned manually.Linux Recovery
Linux is a very flexible operating system, with granular control. This control is demonstrated through the recovery methods available. Another great feature of Linux is the support available from the community and the documentation available. Contact Linuxcertified.com for comments, suggestions or training classes in becoming a more effective system administrator.
About the Author:Ian Smith is a trainer at Linuxcertified.com as well as a consultant in Silicon Valley. Ian also worked on creating a backup appliance with user control, versioning, network backups and easy file restores.
Softpanorama hot topic of the month
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.
ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least
Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info|
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: July, 17, 2017