Sharper shadows for lightmap raytracing?

I am using Embree for Lightmap Generation.
For each Texel in the Lightmap, I shoot rays randomly into the hemisphere of the geometry’s normal.

Vector4 diffuseColor = Vector4.Zero;

RTCRayHit hit = new RTCRayHit();
hit.ray.org_x = sample.WorldPosition.X;
hit.ray.org_y = sample.WorldPosition.Y;
hit.ray.org_z = sample.WorldPosition.Z;
var dir = randomDirHemisphere(sample.Texel.numPathsTraced, sample.Texel.offset, sample.WorldNormal);
float contribution = float.Max(Vector3.Dot(sample.WorldNormal, dir),0);
hit.ray.dir_x = dir.X;
hit.ray.dir_y = dir.Y;
hit.ray.dir_z = dir.Z;
hit.ray.tnear = (32 /1024.0f);
hit.ray.mask = uint.MaxValue;
hit.ray.tfar = 32.0f;
hit.hit.geomID = RTC_INVALID_GEOMETRY_ID;
hit.hit.primID = RTC_INVALID_GEOMETRY_ID;

When a ray hits something, I add it’s color to the texel:

sample.Texel.accumColor += diffuseColor*contribution;
sample.Texel.numPathsTraced++;

When it does not hit anything, I assume it’s the sky:

foreach(var s in suns)
{
                Vector3 rayDir = Vector3.Normalize(new Vector3(hit.ray.dir_x, hit.ray.dir_y, hit.ray.dir_z));
                var nDotL = Vector3.Dot(-rayDir, s.direction);
                nDotL = float.Max(0.0f, nDotL);
                diffuseColor += new Vector4( nDotL* s.color,1.0f);
}

and later, to finish the Lightmap I do:

color = sample.Texel.accumColor / sample.Texel.numPathsTraced;

This works reasonably well actually:

However, the square at the top is actually a hole and the white background is supposed to be the sky.
How would I achieve sharper shadows for the skylight, since it’s supposed to represent the sun? The lightmap’s size is okay, it’s not too small and I checked that.