When a ray of light passes from a less dense to a more dense medium (such as from air to glass) it slows down. If it strikes the glass surface at an angle, it is also bent a little, and this is called refraction. When it passes back into air, it speeds up again, and is again refracted if the surface is at an angle.
Thus a curved piece of glass will focus a parallel beam of light (arriving perpendicular to the lens, that is along its axis) to a point. The interesting thing happens when the light rays are parallel to each other but not parallel to the axis through the centre of the lens. This same lens will also focus these, but to a point above, below, or beside the focal point for rays along the axis, and all these points of focus of parallel rays will form a plane, called the focal plane of the lens. So, you put the sensor at this focal plane, and you've now used the lens to concentrate the light on the sensor.
Focal length is fairly easy to understand with a lens that has a single element, but most camera lenses are made up of lots of separate individual elements. These compound lenses have an effective distance from the image plane, somewhere among all the elements and groups, and the further away from the image plane that is, the longer the focal length. And so when you focus on something closer than infinity, and the lens is moved further away from the sensor (film), the lens will get longer.
This is not the case for all lenses; indeed many have fixed focal lengths that cannot be adjusted. This means that technically a 400mm fixed-focus lens should be 400mm long. But, if you were to get a ruler out, you would see that this is not the case. This is because of all the individual glass inside that makes it behave as if it is longer than it is.
Different compound lenses house varying characteristics - such as the angles at which they refract and disperse light, which ultimately affects image quality.