Imaging & Videochevron_right
Sofwarechevron_right
Hardwarechevron_right
Renderingchevron_right
Algovischevron_right
Computer Graphicschevron_right
HCIchevron_right
linkConvolutional Mask
linkProblem Statement
Apply some convolution mask to images and video
linkBackground
convolution is a mathematical operation on two functions (f and g) that produces a third function (f*g) that expresses how the shape of one is modified by the other. in the case of image processing the convolution is the proccess of adding each element of the image to its local neighbors , weighted by the kernel to extract certain features from an input image.The kernels will define the size of the convolution, the weights applied to it, and an anchor point usually positioned at the center.The origin is the position of the kernel which is above (conceptually) the current output pixel. For a symmetric kernel, the origin is usually the center element.
Of course we are not restricted to 3x3 kernels - this was only done for simplicity. Kernels can be of just about any size. More sophisticated kernels are typically larger, in fact many image processing software packages have options to customize a kernel.
One of the earliest uses of the convolution integral appeared in D'Alembert's derivation of Taylor's theorem in Recherches sur différents points importants du système du monde, published in 1754. Soon thereafter, convolution operations appear in the works of Pierre Simon Laplace, Jean-Baptiste Joseph Fourier, Siméon Denis Poisson, and others. The term itself did not come into wide use until the 1950s or 60s.
linkCode and Results
linkImage
linkConvolution function
First to apply a kernel to a part of an image we use the convolution funtion. This function return the result color of the convolution.
This function is generic, it is useful for any kernel
1link
2linkfunction convolution(x, y, matrix, matrixsize, img) {
3link let rtotal = 0.0;
4link let gtotal = 0.0;
5link let btotal = 0.0;
6link const offset = Math.floor(matrixsize / 2);
7link for (let i = 0; i < matrixsize; i++){
8link for (let j = 0; j < matrixsize; j++){
9link
10link // What pixel are we testing
11link const xloc = (x + i - offset);
12link const yloc = (y + j - offset);
13link let loc = (xloc + img.width * yloc) * 4;
14link
15link // Make sure we haven't walked off our image, we could do better here
16link loc = constrain(loc, 0 , img.pixels.length - 1);
17link
18link // Calculate the convolution
19link // retrieve RGB values
20link rtotal += (img.pixels[loc]) * matrix[i][j];
21link gtotal += (img.pixels[loc + 1]) * matrix[i][j];
22link btotal += (img.pixels[loc + 2]) * matrix[i][j];
23link }
24link }
25link // Make sure RGB is within range
26link rtotal = constrain(rtotal, 0, 255);
27link gtotal = constrain(gtotal, 0, 255);
28link btotal = constrain(btotal, 0, 255);
29link
30link // Return the resulting color
31link return color(rtotal, gtotal, btotal);
32link }
Then to apply the kernel to the whole image we use the function conv(). This function apply an draw the kernel in a new image.
This function is generic, it is useful for any kernel of size 3*3
1linkfunction conv(edgeImg,kernel, posX, posY){
2link edgeImg.loadPixels()
3link
4link for (let x = 0; x <img.width; x++) {
5link for (let y = 0; y < img.height; y++ ) {
6link let c = convolution(x, y, kernel, 3, img);
7link let loc = (x + y*img.width) * 4;
8link edgeImg.pixels[loc] = red(c);
9link edgeImg.pixels[loc + 1] = green(c);
10link edgeImg.pixels[loc + 2] = blue(c);
11link edgeImg.pixels[loc + 3] = alpha(c);
12link
13link }
14link }
15link
16link edgeImg.updatePixels();
17link
18link image(edgeImg, posX, posY, c_w/2, c_h/2);
19link}
linkResult
linkOriginal image
We're going to use the Lenna photos to apply our Kernel.
linkSharpen
Sharpening an image increases the contrast between bright and dark regions to bring out features. We use the following kernel
1linkkernel = [ [ 0, -1, 0 ],
2link [ -1, 5, -1],
3link [ 0, -1, 0 ] ];
4linklet img;
5linklet w = 80;
6linklet c_w = 700;
7linklet c_h = 700;
8link
9linkfunction preload() {
10link img = loadImage('../sketches/lenna.png');
11link
12link}
13link
14linkfunction setup() {
15link createCanvas(c_w, c_h);
16link noLoop();
17link}
18link
19link
20linkfunction draw() {
21link
22link img.loadPixels()
23link
24link SharpenImg = createImage(img.width, img.height);
25link conv (SharpenImg, kernel, 0, 0)
26link
27link}
linkEmboss
Image embossing is a computer graphics technique in which each pixel of an image is replaced either by a highlight or a shadow, depending on light/dark boundaries on the original image. Low contrast areas are replaced by a gray background. The filtered image will represent the rate of color change at each location of the original image. Applying an embossing filter to an image often results in an image resembling a paper or metal embossing of the original image, hence the name.
We use the next Kernel
1linkkernel = [ [ 1, 1, 0],
2link [ 1, 0, -1 ],
3link [ 0, -1, -1] ]
4linklet img;
5linklet w = 80;
6linklet c_w = 700;
7linklet c_h = 700;
8link
9linkfunction preload() {
10link img = loadImage('../sketches/lenna.png');
11link
12link}
13link
14linkfunction setup() {
15link createCanvas(c_w, c_h);
16link noLoop();
17link}
18link
19link
20linkfunction draw() {
21link
22link img.loadPixels()
23link EmbossImg = createImage(img.width, img.height);
24link conv (EmbossImg , kernel, c_w/2, 0)
25link
26link}
linkEdge detection
Edge detection includes a variety of mathematical methods that aim at identifying points in a digital image at which the image brightness changes sharply or, more formally, has discontinuities. The points at which image brightness changes sharply are typically organized into a set of curved line segments termed edges. The same problem of finding discontinuities in one-dimensional signals is known as step detection and the problem of finding signal discontinuities over time is known as change detection. Edge detection is a fundamental tool in image processing, machine vision and computer vision, particularly in the areas of feature detection and feature extraction.[
We use the next Kernel
1linkkernel = 1/9 *[[ -1, -1, -1 ],
2link [ -1, 8, -1 ],
3link [ -1, -1, -1 ] ]
4linklet img;
5linklet w = 80;
6linklet c_w = 700;
7linklet c_h = 700;
8link
9linkfunction preload() {
10link img = loadImage('../sketches/lenna.png');
11link
12link}
13link
14linkfunction setup() {
15link createCanvas(c_w, c_h);
16link noLoop();
17link}
18link
19link
20linkfunction draw() {
21link
22link img.loadPixels()
23link edgeImg = createImage(img.width, img.height);
24link conv (edgeImg, kernel, 0, c_h/2)
25link}
linkBox Blur
A box blur (also known as a box linear filter) is a spatial domain linear filter in which each pixel in the resulting image has a value equal to the average value of its neighboring pixels in the input image. It is a form of low-pass ("blurring") filter. A 3 by 3 box blur ("radius 1") can be written as matrix
We use the next Kernel
1linkkernel = 1/9 *[[ 1, 1, 1 ],
2link [ 1, 1, 1 ],
3link [ 1, 1, 1 ] ]
4link
5linkKernel = [ [0.11, 0.11, 0.11],
6link [0.11, 0.11, 0.11],
7link [0.11, 0.11, 0.11] ]
8linklet img;
9linklet w = 80;
10linklet c_w = 700;
11linklet c_h = 700;
12link
13linkfunction preload() {
14link img = loadImage('../sketches/lenna.png');
15link
16link}
17link
18linkfunction setup() {
19link createCanvas(c_w, c_h);
20link noLoop();
21link}
22link
23link
24linkfunction draw() {
25link
26link img.loadPixels()
27link BlurImg = createImage(img.width, img.height);
28link conv (BlurImg , kernel, c_w/2, c_h/2)
29link}
linkAll in one
To compare all the effects we put it together
linkVideo
For video we can get pixels periodically from de canvas and make the convolution with the kernels, in this case we only show three frames by second. Hence we paint a new image with the result of the convolution under the original video. Click on the canvas for play the video and next on the button for change the kernel and see the new convolution:
1linklet fingers;
2linklet lienzo;
3linklet contador=0;
4linklet reproduce = false;
5link
6linkconst Blur_Kernel= [ [0.11, 0.11, 0.11],
7link[0.11, 0.11, 0.11],
8link[0.11, 0.11, 0.11]];
9linkconst Border_Detection= [ [ -1, -1, -1 ],
10link[ -1, 8, -1 ],
11link[ -1, -1, -1 ] ];
12linkconst Emboss= [ [ 1, 1, 0],
13link[ 1, 0, -1 ],
14link[ 0, -1, -1] ];
15linkconst Sharpe= [ [ 0, -1, 0 ],
16link[ -1, 5, -1],
17link[ 0, -1, 0 ] ];
18link
19linklet matrixCarrousel=[Blur_Kernel,Border_Detection,Emboss,Sharpe];
20linklet matrix =matrixCarrousel[0] ;
21linkconst matrixsize=3;
22linkfunction setup() {
23link fingers = createVideo(['/vc/docs/sketches/fingers.mov', '/vc/docs/sketches/fingers.webm']);
24link fingers.hide();
25link createCanvas(640, 240);
26link lienzo=createImage(320,240);
27link button=createButton('Change Kernel!');
28link button2=createButton('Play/Pause');
29link button.position(275,200);
30link button2.position(290,225);
31link button.mousePressed(changeMatrix);
32link button2.mousePressed(pauseVideo);
33link frameRate(3);
34link noLoop()
35link}
36link
37link
38linkfunction draw() {
39link background(0);
40link lienzo.loadPixels();
41link fingers.loadPixels();
42link
43link for (let x = 0; x <fingers.width; x++) {
44link for (let y = 0; y < fingers.height; y++ ) {
45link let c = convolution(x, y, matrix, matrixsize, fingers);
46link let loc = (x + y*fingers.width) * 4;
47link lienzo.pixels[loc] = red(c);
48link lienzo.pixels[loc + 1] = green(c);
49link lienzo.pixels[loc + 2] = blue(c);
50link lienzo.pixels[loc + 3] = alpha(c);
51link }
52link }
53link lienzo.updatePixels();
54link image(fingers,0,0);
55link image(lienzo,fingers.width+1,0);
56link}
57linkfunction changeMatrix(){
58link contador=(contador+1)%matrixCarrousel.length;
59link matrix=matrixCarrousel[contador];
60link}
61link
62linkfunction pauseVideo() {
63link reproduce = !reproduce;
64link if (reproduce){
65link
66link loop();
67link fingers.loop(); // al presionar en el lienzo blanco inicia el video
68link }else{
69link noLoop();
70link fingers.pause(); // al presionar en el lienzo blanco inicia el video
71link }
72link}
linkReference
Filter taking from: