4V is not high enough. The USB spec requires 5V +/-5% and many USB devices won't draw power until the voltage is correct and the data line voltages are correct.
Test your panel in sunlight. Measure the open circuit voltage and the short circuit current, if you have a digital multimeter or equivalent.
If you have sufficient voltage, you will need to add a 5V regulator and bias the data pins with some resistors. Scroll down to the April 27, 2006 post by ranjeevm and use the upper diagram.
http://www.instructables.com/id/How-To-Make-Your-Own-USB-Car-Charger-For-Any-iPod-/
Your panel will need to provide sufficient current (and power). Here is an article where power draw was measured. It appears you typically will need at least 6W there-abouts. If your panel is undersize, your results may be less than satisfactory:
Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 G, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 S, NEW 2600mAh Solar panel USB Power Battery Charger for CellPhone MP3 MP4 Camera, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 P, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 P, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 G, USB2.0 Solar Battery Panel Charger for Cell Phone MP3, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, Brunton Solaris 2 USB Foldable Solar Panel w/ Solo 3.4 Lithium Mini Storage , Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 U, 4440mWh USB Powered Solar Panel Charger for Cell Phone, SOLAR PANEL KIT INC USB FOR CHARGING DEVICES AND HAS A 14 LED LIGHT BL-SPK1, NEW HOT USB 2600MAH Solar Battery Panel USB Charger for MP3 Cell Phone PDA, Solar Panel Battery Charger for Samsung Galaxy Tab CDMA/T-Mobile+USB Data Cable, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 S, Emergency 2600mAh Solar panel USB Battery Charger For CellPhone Rechargeable gw, NEW Solar Panel USB Battery Charger for mobile cell phone nokia Samsung MP4 PDA, Solar Panel USB Battery Charger for mobile cell phone nokia Samsung MP3 MP4 PDA, 2600mAh Solar Panel Powered Back Up Battery USB Charger for iPhone 4/4S iPad 2, Emergency 2600mAh Solar panel USB Battery Charger For CellPhone Rechargeable gw, Emergency 2600mAh Solar panel USB Battery Charger For CellPhone Rechargeable gw, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 P, NEW 2600mAh Solar panel USB Power Battery Charger for CellPhone MP3 MP4 Camera, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, Emergency 2600mAh Solar panel USB Battery Charger For CellPhone Rechargeable gw, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, 16000mAh Solar Panel USB Charger Battery for Cell Phone Laptop Notebook Tablet, Portable Multi-function USB Solar Battery Panel Charger for Cell Phone MP3 MP4 U, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, USB Solar Battery Panel Charger for MP3/MP4/Cell Phone, PowerFilm USB+AA-PowerFilm USB + AA Solar Panel Charger - Kit, Universal 3600mAh Solar power Panel USB Battery Charger for mobile phone PDA, New Universal 3000mAh 60led Solar Panel USB Battery Charger for mobile phone PDA, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, Emergency 2600mAh Solar panel USB Battery Charger For CellPhone Rechargeable gw, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, Flexible Solar Panel 3watt camoflage USB Port for charging, SALE Solar Battery Panel USB Charger for MP3 Cell Phone PDA iPhone 4s 4 3Gs, 16000mAh USB Powered Solar Panel Charger for Cell Phone, 2600mAh Solar panel USB Battery/Charger 4 CellPhone gw, UK Portable 2600MAh USB Solar Battery Panel Charger for Mobile Phone MP3 MP4 PDA,
