It might sound crazy but I believe it to be an extremely valid point. I've read a fair few books in recent months that could easily have had a chapter removed, or sometimes two or three early chapters condensed into one. I might even go as far as to say I could remove chapter one from most books I've read since starting this site and rarely would I miss out on anything salient.
Now, I'm not suggesting you should go and remove your first chapter without delay, but try re-reading it and consider what information is contained within it that is important to the reader. Simply saying “it sets the scene nicely” or “it introduces and tells you a bit about the main character” aren't reason enough for its inclusion. Could you let the reader get to know the main character through his or her actions in chapter two or three? Could you describe the scene or location within the context of the action further along in the book?
If you're telling the story of the investigation of a kidnapped girl, is it really necessary to detail the kidnapping in chapter one? If the point of the book is the investigation, it doesn't really matter how the kidnapping was staged or who did it. Why give the reader an advantage over the detective? Why not let them be in the dark together, at least initially? Don't start the story with your detective getting out of bed, making coffee, having breakfast, talking to his wife and packing the kids of to school, start it with him walking past his superior's office and getting shouted at. You can make it clear through this interaction that your detective hasn't slept well, is in need of his morning coffee, has milk stains on his tie where one of his kids had spilt their cereal etc.
Sometimes it helps to write the first chapter as a storyboard exercise, for your own benefit as the writer. It's often easier to write chronologically, so if it helps to write that 'morning scene' you should. Not everything you commit to paper should end up in the finished book though. Cutting things out is an important part of the editing process. Removing scenes that don't contribute to the plot or character development, sometimes even removing whole characters and giving their important contributions to other existing characters.
When you sit with your first draft of your manuscript, be aware of how critics, agents, publishers and eventually customers will view your first chapter. If it moves briskly, introduces the main character, sets the scene and reveals something tantalising at its end, you've done your job of enticing the reader. In the modern world of eBooks with free samples it's become more important than ever to get to the point quickly and effectively. I love nothing more than reading one chapter of a book and knowing I want to read on. Sometimes it feels like authors are doing everything in their power to stop me from turning that page!
So, be concise, get to the point, leave the reader in no doubt that they have to read chapter two.