Debugging tips for your CGI scripts

Updated Dec 29 at 09:34 CDT (first posted Sep 21 at 05:05 CDT) by Remi in CGI script, FAQ, Help, Python, Software  - 15 comment(s)

• Symptoms
• Put on your detective hat
• A working script
• Debugging tips

Symptoms

So you've carefully written your CGI script, you transfer it to our servers and bang, it doesn't work !

You most likely get the dreaded "Internal server error" in your browser and if you look at the error logs in Apache, you probably see the infamous "Premature end of script headers".

What the heck is happening ?

The "Internal server error" in your browser and the "Premature end of script headers" message in the logs are Apache's way of telling you "There's something wrong with your script" ... Very helpful isn't it ... But don't panic, there are several things you can do to see what's causing the problem.

Put on your detective hat

Check the permissions

Regardless of which user will execute the script (either the "apache" user, or yourself if Apache is configure with SuExec - which is the case at WebFaction), that user will need the executable permission on the script *and* the directory. But that's not all ! With Apache there is such a thing as "too many permissions" ... This seemed very counter-intuitive to me at first and the only interpretation that I can think of for this is that Apache is basically thinking "this is not safe enough, I'm not going to run that script". In general, you can use 711 as the permissions for the script and the directory:

[test7@web3 htdocs]$ chmod 711 . test.py [test7@web3 htdocs]$ ls -al total 12 drwx--x--x 2 test7 test7 4096 Sep 20 05:56 . drwxr-xr-x 5 root root 4096 Sep 20 05:51 .. -rwx--x--x 1 test7 test7 88 Sep 20 05:56 test.py [test7@web3 htdocs]$

Try running the script from the command line

If the script doesn't run from the command line, it's not going to work from the browser. Here are a few things that can happen when you run it from the command line:

1. "bad interpreter: No such file or directory"

As seen here:

[test7@web3 htdocs]$ ./test.py : bad interpreter: No such file or directory [test7@web3 htdocs]$

This is a common error and it usually happens when you transfered a file from Windows to Linux and didn't convert the "\r\n" into "\n". In that case, there is an extra character (invisible from a normal editor) on your shebang line and it's confusing the shell. One way to check if this is the case is to use the "hexdump" tool to see the exact bytes in your text file:

[test7@web3 htdocs]$ hexdump test.py 0000000 2123 752f 7273 6c2f 636f 6c61 622f 6e69 0000010 702f 7479 6f68 326e 352e 0a0d 0a0d 7270 ... [test7@web3 htdocs]$

Notice the "0a0d" sequence ... This means that your lines end with "\r\n". In order to replace that with just "\n" you can either tell your ftp or sftp client to do the conversion for you, or you can use the "dos2unix" tool to do that for you:

[test7@web3 htdocs]$ dos2unix test.py dos2unix: converting file test.py to UNIX format ... [test7@web3 htdocs]$ hexdump test.py 0000000 2123 752f 7273 6c2f 636f 6c61 622f 6e69 0000010 702f 7479 6f68 326e 352e 0a0a 7270 6e69 ... [test7@web3 htdocs]$

Notice how the "0a0d" is gone and it's just "0a" now. Your script should work fine now.

2. "xxxxx: bad interpreter: No such file or directory"

As seen here:

[test7@web3 htdocs]$ ./test.py -bash: ./test.py: /usr/local/bin/python2.6: bad interpreter: No such file or directory [test7@web3 htdocs]$

This error means that the path that you specified on your shebang line doesn't exist. In the example, "/usr/local/bin/python2.6" doesn't exist. If I replace it with "/usr/local/bin/python2.5" then my script works fine.

A working script

Sometimes it's good to have a sample working script so that you have a good starting point to write your other scripts. Here is a sample script (and its permissions) that works:

[test7@web3 htdocs]$ more test.py #!/usr/local/bin/python2.5 print "Content-Type: text/plain" print print "Hello, world" [test7@web3 htdocs]$ ls -al . total 16 drwx--x--x 2 test7 test7 4096 Sep 21 05:22 . drwxr-xr-x 5 root root 4096 Sep 20 05:51 .. -rwx--x--x 1 test7 test7 88 Sep 21 05:26 test.py [test7@web3 htdocs]$ ./test.py Content-Type: text/plain Hello, world [test7@web3 htdocs]$

Debugging tips

Checking if the script runs at all

Sometimes it's not even clear if Apache runs the script at all.

One simple trick you can use to check if the script is running at all is to make it create a file in /tmp from the very beginning:

[test7@web3 htdocs]$ more test.py #!/bin/sh touch /tmp/foo [test7@web3 htdocs]$

You can then try to request the URL in your browser and if your script runs at all you should see a file called "foo" in /tmp.

Note that you can use a similar technique for writing debugging information from your script to a file.

Using the error logs

Anything your script prints on stderr will go in your Apache's error logs. For instance, if you use a python script and an error happens in your script, you will see the traceback in your Apache's error logs.

You can also manually write to stderr from your script and check the messages later from your logs. Here is an example:

[test7@web3 htdocs]$ more test.py #!/usr/local/bin/python2.5 import sys sys.stderr.write('Made it here\n') print "Content-Type: text/plain\n" print print "Hello, world" sys.stderr.write('Made it there\n') a = b # Error sys.stderr.write('Made it to the end\n') [test7@web3 htdocs]$

Then you can request the URL in your browser. And now check the logs:

[test7@web3 htdocs]$ tail ../../logs/apache/error_test7.log ... [Thu Sep 21 05:43:18 2006] [error] [client 86.137.209.68] Made it here [Thu Sep 21 05:43:18 2006] [error] [client 86.137.209.68] Made it there [Thu Sep 21 05:43:18 2006] [error] [client 86.137.209.68] Traceback (most recent call last): [Thu Sep 21 05:43:18 2006] [error] [client 86.137.209.68] File "test.py", line 9, in <module> [Thu Sep 21 05:43:18 2006] [error] [client 86.137.209.68] a = b # Error [Thu Sep 21 05:43:18 2006] [error] [client 86.137.209.68] NameError: name 'b' is not defined [test7@web3 htdocs]$

15 comments:

---

usually I use
tail -f [filename]
and leave that running in another shell window.
that way I can debug the script in one window, and see the errors as I go along.. much faster

---

Ok, so far so good. But you assume that once you get a script working on the command line, it'll work from a remote browser.

Say I've got a script printing:

$ ./test.cgi
Content-Type: text/plain

Hello, World!

but navigating to that script's URL still gives a 500. What then? (Because I'm actually experiencing this exact error.)

---

Ok, here's more detail:

I've got a file, index.cgi, in directory "upload". I've set chmod 711 on index.cgi in that folder. Spits out desired HTML when I run it from an SSH window. However, pointing my browser to http://www.lillimuse.com/upload redirects to http://www.lillimuse.com/upload/upload (???) though there's no file by that name in the folder, and bypasses index.cgi, so I get a 404. Browsing to upload/index.cgi does the same thing - attempts to load upload/upload. Thoughts? Thanks!

---

Ooops, disregard above comments. It was my fault - remember to chmod 711 your directory, as the FAQ says, before you go posting comments in distress!

---

Thanks for the info, I don't think I would've ever got my first .py off the ground without this info. Ok, I lie, but it definitely would've taken me quite a bit longer.

Also, my preferred text editor, Edit+, lets you specify (PC|Unix|Mac) encoding for files, which is really helpful for development :)

---

Thanks for the info, I don't think I would've ever got my first .py off the ground without this info. Ok, I lie, but it definitely would've taken me quite a bit longer.

Also, my preferred text editor, Edit+, lets you specify (PC|Unix|Mac) encoding for files, which is really helpful for development :)

---

Thank you so much for this web page! It was a great help for me :D

---

My Perl script outputs HTML from Webfaction's command line but outputs the Perl code when I access the file via http. The server doesn't execute the code but just outputs the plain text of the Perl file. I've played with the permissions as described above, and the shebang line points to Perl correctly. Any idea why it's not being executed?

---

Tim: Does your script have a ".cgi" extension ? Right now only .py and .cgi files are treated as cgi scripts.

---

Remi: That solved it! My Perl files had a .pl extension; I changed them to .cgi and they work.

---

So I made a basic test.cgi test script, which runs fine from the shell:
-----BEGIN-----
#!/usr/local/bin/python2.5

print "Content-Type: text/plain"
print "Hello, world"
-----END-----

However, I get the ol' 500 error and Premature end of script headers in the log when I try run from the browser.
I have chmod'd both the directory and my test.cgi

Any suggestions on where to look next?

Thanks

---

josh: you're missing one "print" statement (or you need a \r\n after your first string) :)

Remi.

---

d'oh. right you are. Thanks. Now to get the more interesting script running...

---

I keep getting Script errors on line 56 and 30 whats ups and how can I fix it???

---

Hello Beth,

You should ask your question in our forum or one of the language that you wrote the script in's mailing lists. No one will be able to help without seeing lines 30 and 56 of your script, at least, not without playing a game of twenty questions. :)

---

Leave a new comment:

(Note: comments may be moderated)

Author:
Email (not displayed):
Website:
Message: