DIGITAL PHOTOGRAPHY NEWS

Sony Developing 60 fps CMOS Image Sensor and DSP for Video and Still Cameras

High-Speed CMOS Sensor and Camera Systems Creating a New Imaging World

April 21, 2006 -- Starting with their adoption in cellular phones in 2002, Sony's CMOS sensors for megapixel class camera cellular phones, digital high-definition video cameras, and Digital Single Lens Reflexs (DSLRs) have become widely used in commercial products. The greatest strength of CMOS sensors is that by including both digital and analog circuits on the same chip, their readout speed can be made dramatically faster than that of CCD. Sony is now pushing forward with the development of high-speed CMOS sensors for digital cameras that can provide continuous imaging from all pixels at the high frame rate of 60 frame/s. Sony is also developing in parallel a camera digital signal processor (DSP) that can take full advantage of this performance. Sony is now proposing a new way of enjoying photography in which one photographs and records images without concern for the difference between moving images and still images by using this frame readout of 60 frame/s high-speed CMOS sensor and new camera DSP.

Life You Wanted to Capture. Moments You Missed.
Recently, the trend towards higher pixel counts in consumer digital cameras has continued, and it is now possible to take high-resolution still images conveniently. Sony has, until now, focused its efforts in the image sensor area on increasing the pixel count (shrinking the size of the unit pixel)and improving the pixel's characteristics to achieve high-resolution still imaging and has contributed to increasing the popularity of digital cameras. We feel, though, that we can say that we have now achieved a level higher than just capturing "pretty" images. We think that from here on, what will become important for the further development of digital cameras is evolving from recording still images to the digital camera being a tool for communication and self expression using moving images.

One theme for that evolution that we can present is high frame rate imaging at high pixel counts. In particular, if it will become possible to shoot continuously at 60 frame/s at a high pixel count, while the individual images that make up the moving image will have the quality of still images and the dividing line between still and moving images will blur. Since the user can take moving and/or still images without concern for the difference, a previously unavailable expressive power will become available. Although the message delivered by a single still image is not large, by interweaving that still images with moving images, the amount of information that can be transmitted increases dramatically, and a strong impact can be delivered to the viewers. Furthermore, choosing the one "that's it!" image from the 60 frame/s of high-resolution images captured in a second allows one to find the perfect shutter moment after the fact.

Sony, in aiming for an image sensor that can capture and express surprise and emotion, is now putting its efforts into the development of both a CMOS sensor that can express, in particular, motion and the beauty of a moment and a camera DSP that can get the full performance from that sensor. Sony's theme for this effort is capturing "Life you wanted to capture. Moments you missed." Sony is committed to contributing to this new imaging world.

Capturing the "Perfect Shot" from a Moving Picture
Up to now image sensors have been able to read out the entire frame at only a few frames per second. Additionally, when switching to moving picture mode, a few frames at the beginning and the end will not be output correctly. Therefore, a user trying to take that "perfect shot" while capturing moving picture would lose that shot and the moving picture would be stopped for a few frames. Even if the user succeeded in getting that "perfect shot", the critical scene would be missing from the moving picture. So users have to give up on capturing moving picture for the scene if they want to capture a still shot.

If, however, still images are captured seamlessly at 60 frame/s during moving picture, it would be possible to dramatically increase the flexibility of imaging. The user could, without concern for any limitations on imaging, freely record moving picture that is no interrupted for stills, and capture the scenes before and after a still image without interrupting the moving picture. For example, if your child hits a winning home run in the bottom of the ninth, you would not be limited to a single still photograph, but could see that image woven into the continuous moving picture. Were this possible, the feeling of live performance, dynamism, and import of that single image would be amplified. We hope this ability dramatically increases the impact that photographs have on people.

This functionality would be effective at a variety of events, such as parties and wedding receptions. Furthermore, raw footage of mixed still and moving images acquired together can be processed with software to create TV commercials or promotion video class productions and expressions, even from ordinarily photographed scenes.

High Picture Quality that Captures the Moment
Although image quality improves as you increase the amount of information captured, until now, still image picture quality has been improved by increasing the spatial resolution. However, the amount of information can be increased not only by increasing the spatial resolution, but by increasing the amount of information in the time direction as well.

Picture quality can be greatly increased by increasing this time direction information.

Multiplane Addition
Recently, the number of users who want to shoot indoors or evening/night scenes cleanly and preferably without flash has been increasing. Thus cameras that boast camera shake correction or high ISO sensitivity have appeared in the market. These products adopt either camera shake correction functions in the lens or high-sensitivity image sensors.

However, there is another way to achieve camera shake correction and higher sensitivity (higher Signal-to-noise ratio). This method consists of continuous imaging of the entire frame at 60 frame/s, and using this increased amount of information in the time direction to create a single high-quality image. This can be implemented in a total camera system that includes a camera DSP. For example, if someone applies multiplane addition to several images that were captured at 60 frame/s, an image with a signal-to-noise ratio several times better can be acquired. Also, a high-speed camera shake correction function can be implemented if images are stacked while applying camera shake correction to each image. This function can make it possible to capture bright, camera-shake-free images even without an inlens camera shake correction function even in slightly darker environments where camera shake can easily occur, such as school festivals, children's plays or presentations, or indoor events.

The More Clearly You See Life, the Better It Gets
People are often surprised and moved by images and moments that normally cannot be seen. Moving picture at high frame rates, that is frame rates far in excess of 60 frame/s, can arouse our sleeping curiosity and lead to new, previously unknown, surprises and discoveries. The high-speed imaging achieved by high-speed CMOS sensors also have the possibility for creating new cameras. The world of high-speed imaging, which was previously only used for special applications by organizations such as TV stations, can now be implemented easily for consumer products is truly revolutionary.

There are many situations where it can be fascinating to play back high-speed video in slow motion. These include checking a golf or baseball swing, reviewing your children's sports or a goal they made at a school sporting event with the family. Or, by academics in various fields to find out how birds take off from the water, how a water balloon breaks, or how other natural phenomena occur. (See figure 5.) If you add appropriate background music, these slow-motion videos can be truly moving. If we modify the CMOS sensor readout method slightly, it will be comparatively easy to create high-speed imaging at over 60 frame/s. Sony is working to create high-speed CMOS sensors that can easily capture events that exceed the capabilities of the human eye.

Future Developments
Sony has developed image sensors that take the utmost advantage of the high-speed characteristics of Sony?s CMOS technology for applications such as hard disk video recording and seamless capture of moving and still images. Sony is also aiming at achieving high-speed imaging in the several hundred frame/s range and is searching for possibilities for new types of digital cameras. Sony is aiming to increase the number of pixels that can be read out at 60 frame/s and to achieve even higher speed imaging to contribute to a new axis of high-quality multifunctional digital camera products.

By combining the pixel design technologies fostered in Sony's CCD work with Sony's CMOS analog and digital circuit technologies, Sony delivers products that lead to even further productivity in the imaging world. Keep your eye on Sony for even more exciting developments in our CMOS sensor area.