Check and start a process in Linux
Over the past few days, we realised that, for various reasons, our Sphinx search process (searchd) would occasionally not be running. As Sphinx is pretty much the backbone of our sites, we couldn't have this happening. I wrote a small script that runs permanently, checks if searchd.pid exists (you can have it check anything) and if not, then starts searchd. This is all done in a screen session so I can just leave it running.
The script is very, very simple;
while [ "1 -eq 1" ]
do
if [ ! -e "/var/log/searchd.pid" ]
then
searchd
fi
sleep 2
done
Basically, we have an infinite loop, we check that /var/log/search.pid exists, if not, start it, then sleep ('pause' essentially) for 2 seconds before starting again. You can change it to alert you too, if you'd like.
To get an idea of how screen works, check out this post.
Thought it might help some of you.
What happened to all my hard drive space?
Recently I was trying to upgrade using Yum on my Fedora 9 machine and I got an awful message that I did not have enough space to complete the upgrade. What the hell? My / partition has a huge 10GB, how can this be so?
[root@Garth Download]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.5G 8.7G 312M 97% /
tmpfs 722M 80K 722M 1% /dev/shm
/dev/sda4 17G 12G 4.2G 75% /home
/dev/sda1 28G 25G 3.1G 90% /mnt/Winxp
As you can see, my laptop has a 60GB hdd, so I am consistently having this argument with it.
The problem with most command line utilities is that it is difficult to see what you have installed and how much space each has. Then I remembered a little script that I used a few years back that told me what RPM's I had installed and how big each was. Eureka! Just what I need.
So, I had a little fun, did a little search and found the beloved rpmhogs script.
Check this out;
[root@Garth Download]# perl rpmhogs.pl
246,007,784 java-1.5.0-gcj-javadoc-1.5.0.0-21.fc9.x86_64
246,902,300 java-1.6.0-openjdk-javadoc-1.6.0.0-0.13.b09.fc9.x86_64
231,068,455 openoffice.org-core-2.4.1-17.4.fc9.x86_64
114,684,216 texlive-texmf-fonts-2007-22.fc9.noarch
91,189,935 glibc-common-2.8-3.x86_64
87,927,623 compat-gcc-34-c++-3.4.6-9.x86_64
84,367,402 git-1.5.5.1-1.fc9.x86_64
<snip>
How good is that? I love it. Now I need to uninstall a crap load of stuff I dont use.
And, yes, my Laptop's name is Garth. My media PC is named Cassandra and my wifes laptop is named Wayne. Who can guess my theme?
The common grep –exclude problem
grep is a file content searching tool for Linux. It is loved by programmers for its ease of use and swift search results (with customisable output). For example, if you forgot where you left your debug code, you can quickly display a list of the files that contain the code like so;
grep -ril 'var_dump' *
But, the problem with grep is that it is not possible to limit the subdirectories it searches within based on a pattern. A lot of people misunderstand the "-exclude" option, thinking that they can stop grep from returning results from SVN directories using it like so;
grep -ril --exclude=".svn/*" 'var_dump' *
Little do they realise that exclude is for file names only and grep has no functionality for restricting search based on directory names. grep can invert the found rows and only show the non-matching lines using the '-v' parameter and can remove the .svn directories but it is very slow;
grep -ril "'var_dump' * | grep -v svn
So, in comes ack. ack is much faster than grep (it uses perl regular expressions) and searches recursively and ignores revision control subdirectories (.svn) by default. Also, it displays the found code in a much easier to read format (if you opt to display the lines as opposed to just the file names);
To make things easier, ack is available in a lot of repo's for different Linux distributions. I was able to get ack this easily;
sudo apt-get install ack
Done. Its also available as a perl script incase you cannot install it due to dictator sys admins (you know the type.).
If you code and use grep for your line-level searching, do yourself a favour and get ack.
ER Diagrams from SQL files
A lot of people these days use graphical clients for creating their database schema's and therefor start with the Entity Relationship diagram and then have it generate the SQL for them. I prefer to write SQL directly as I seem to have an ongoing misunderstanding with most graphical clients but then I would need to somehow create an ER diagram for the SQL I wrote and this meant I'd double up on work a little bit. Today while searching for another application to do just what I want (and not cost me a cent) I found a post about something that will be extremely handy.
SQL Fairy is an ER Diagram creator for Mysql and other databases but uses the raw SQL files and includes relationships (like foreign keys etc).
Its not perfect by any means, but its easy and gives a great representation of the database relationships. The lines representing relationships actually go over the tables which is a bugger and as it outputs directly to an image, you cant edit it. But, for a quick and easy ER, its bloody good.
As noted in the post I found this little gem at, its super simple to install and use, particularly if you are using Linux. I am on Kubuntu (for the record, my distro of choice is Fedora) and installed it as easy as this;
sudo apt-get install sqlfairy
Then, to generate the diagram was as easy as this;
sqlt-diagram -d=MySQL -o=sample_schema.png schema.sql --color
Unfortunately, it did fall over when my schema contained a SOURCE within the SQL, but apart from that it was quick and simple enough for me.
Now, if I can find an app that does the same thing but allows me to edit my ER first and give me multiple saving options (XML, SQL, etc) then I'd be extremely happy.
sqlt Man page
sqlt-diagram Man page
EDIT: About a year ago, MySQL released an application named 'Workbench' for this very purpose. An ER diagram program with reverse engineering capabilities from .sql files. The application was used by me on Windows for a while before a Linux friendly version was released. But, when Sun bought MySQL earlier this year, they made most of the MySQL applications commercial and were charging for licenses. Thats why Workbench was not mentioned above.
I am pleased to announce that it appears Workbench is free for download again. More information on Workbench can be obtained at its website.
This is the same schema as the above diagram done in Workbench using reverse engineering from an SQL file.
Debian etch apt-get issue
I am setting up a local development server at work as our main development server is in Melbourne and we have no root access (so permissions changes on directories can take days, literally, before they are actioned). I am using debian as it is the OS used on the main dev server (if it was up to me, I’d use CentOS, but thats a story for another day). I used the internet setup disc image to minimise what I downloaded and installed a bare-bones system on a machine here.
When I went to use aptitude (apt-get) to install required software and servers, I kept getting errors advising me of the dependants needed for the install, but the error says “but it is not installable”. For some reason, I was unable to install anything. Anyway, after some searching it appeared that the default sources.list file (/etc/apt/sources.list) contained repositories for security patches only. Adding the following line sorted it for me; (NOTE. Use a locally hosted mirror or updates will take you forever);
deb http://mirror.pacific.net.au/debian etch main
Also, as a side note, on etch, use vim, not vi as its much, MUCH easier and you wont pull your hair out as much.
Code merging. Its awful without good tools.
No developer likes to merge their code in with other code after finishing a big project. Its a royal pain in the ass. I find myself having to do it at work a lot (we have a largish team) and at home because at home, I somehow find that I have 84 copies of the same thing. So, I need to merge them together occasionally to keep things maintainable.
Merging is by far 10 times easier on Linux than windows. Linux is such a good development environment (apart for .net – but then, who uses .net anyway?). I would recommend getting VirtualBox, Fedora (or Ubuntu) and installing them on a windows machine. Even if you only use it for coding, profiling and merging / deploying. Its worth it and takes bugger all time to set up.
I am using Ubuntu on my VirtualBox. I prefer Fedora, but I found it hard to get higher resolutions with it. Anyway, I’m off topic.
Once you create your two directories or files that you need to merge or create a diff for. You can open one of the three programs;
My personal favourite is Meld. There are others out there. If you use something else, let me know about it.
Using one of these apps takes the hard work out of merging. In meld, you can match custom rules (for example, you can match all files in the ‘cache’ directory) and then ignore them.
Then, once your files / directories are merged, you can use something like KDESvn to commit them (if you are not savvy enough with the command line SVN client).
All done. Merged, Committed and time for a beer.
If you are unlucky to be on a Windows machine. No real biggy, but your choice of free clients becomes slim.
TortoiseSVN is the app of choice for interacting with an SVN repo.
Once you have the directories, you can merge them with WinMerge. I do find this app a little lacking. It doesn’t display the files in a heirarchical view, so diffing 10,000 files (which I did yesterday) just lists them all one by one and not within their directories.
I have heard a lot of good things about Beyond Compare but it is not free.
Let me know what you use for comparing, diffing and merging files.
Using lftp to transfer a large site
I own and run a fairly large site (www.skylinesaustralia.com) and have recently decided to give up the dedicated server it has been running on and move to a managed solution with mosso.
The problem for me was that mosso do not yet have ssh access due to their network config and this meant that I would need to transfer SAU over FTP from my server. (6.5GB uncompressed database and 60GB file structure meant that this wasn’t going to be pretty).
For anybody that has dabbled in Command Line FTP transfers, its not very pleasant. FTP doesnt natively support recursive transfers (this is performed at the client level) so transferring my site non recursively would have been painful for even the most boring-job lovers. This coupled with the fact that it does not offer any form of synchronising incase the connection drops (which occurs a lot in ftp) or if we needed to transfer over several sessions. Enter lftp. This extremely cool little ssh based app is like an ftp client on steroids.
Now, these instructions are for a rhel based OS (I am on CentOS 4.5) but wouldn’t be hard to make them work on other OS’s.
So, to start, we will need lftp installed;
$ yum -y install lftp
Now, lftp has an awesome ‘mirror’ feature which does just that, mirrors one directory to another. We can also specify to only transfer newer files so we can launch the transfer as many times as we want and it will only transfer changed / new files (like rsync).
So, we launch lftp like so;
$ lftp -u username,password –e “set net:timeout 0” –e “mirror --reverse –verbose -n /local/directory remote/dir” ftp.yourftpserver.com
This does the following;
- -u specifies the username and password for the server you are connecting to
- -e allows you to ‘queue’ commands. So, instead of waiting for lftp to be ready for the next command, we can write it all in one statement and it will just run them in order.
- The first command is “set net:timeout 0” – this is just to disable the timeouts with lftp. Some of my directories contained hundreds and hundreds of subdirectories and it would timeout when reading them.
- The second command is the good one, its for mirroring two directories and its commands are;
- -revserse just switches the order of the directories (ie local remote, not remote local)
- -verbose just displays what lftp is doing
- -n copies only new files
- The final argument is the remote ftp server.
This has worked flawlessly for me several times now and it is the only command line ftp method I use. The standard ftp client on *nix boxes is ridiculous.
Let me know how you go with running this.