function [x,sNorm] = som_norm_variable(x, method, operation) %SOM_NORM_VARIABLE Normalize or denormalize a scalar variable. % % [x,sNorm] = som_norm_variable(x, method, operation) % % xnew = som_norm_variable(x,'var','do'); % [dummy,sN] = som_norm_variable(x,'log','init'); % [xnew,sN] = som_norm_variable(x,sN,'do'); % xorig = som_norm_variable(xnew,sN,'undo'); % % Input and output arguments: % x (vector) a set of values of a scalar variable for % which the (de)normalization is performed. % The processed values are returned. % method (string) identifier for a normalization method: 'var', % 'range', 'log', 'logistic', 'histD', or 'histC'. % A normalization struct with default values is created. % (struct) normalization struct, or an array of such % (cellstr) first string gives normalization operation, and the % second gives denormalization operation, with x % representing the variable, for example: % {'x+2','x-2}, or {'exp(-x)','-log(x)'} or {'round(x)'}. % Note that in the last case, no denorm operation is % defined. % operation (string) the operation to be performed: 'init', 'do' or 'undo' % % sNorm (struct) updated normalization struct/struct array % % For more help, try 'type som_norm_variable' or check out online documentation. % See also SOM_NORMALIZE, SOM_DENORMALIZE. %%%%%%%%%%%%% DETAILED DESCRIPTION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % som_norm_variable % % PURPOSE % % Initialize, apply and undo normalizations on a given vector of % scalar values. % % SYNTAX % % xnew = som_norm_variable(x,method,operation) % xnew = som_norm_variable(x,sNorm,operation) % [xnew,sNorm] = som_norm_variable(...) % % DESCRIPTION % % This function is used to initialize, apply and undo normalizations % on scalar variables. It is the low-level function that upper-level % functions SOM_NORMALIZE and SOM_DENORMALIZE utilize to actually (un)do % the normalizations. % % Normalizations are typically performed to control the variance of % vector components. If some vector components have variance which is % significantly higher than the variance of other components, those % components will dominate the map organization. Normalization of % the variance of vector components (method 'var') is used to prevent % that. In addition to variance normalization, other methods have % been implemented as well (see list below). % % Usually normalizations convert the variable values so that they no % longer make any sense: the values are still ordered, but their range % may have changed so radically that interpreting the numbers in the % original context is very hard. For this reason all implemented methods % are (more or less) revertible. The normalizations are monotonic % and information is saved so that they can be undone. Also, the saved % information makes it possible to apply the EXACTLY SAME normalization % to another set of values. The normalization information is determined % with 'init' operation, while 'do' and 'undo' operations are used to % apply or revert the normalization. % % The normalization information is saved in a normalization struct, % which is returned as the second argument of this function. Note that % normalization operations may be stacked. In this case, normalization % structs are positioned in a struct array. When applied, the array is % gone through from start to end, and when undone, in reverse order. % % method description % % 'var' Variance normalization. A linear transformation which % scales the values such that their variance=1. This is % convenient way to use Mahalanobis distance measure without % actually changing the distance calculation procedure. % % 'range' Normalization of range of values. A linear transformation % which scales the values between [0,1]. % % 'log' Logarithmic normalization. In many cases the values of % a vector component are exponentially distributed. This % normalization is a good way to get more resolution to % (the low end of) that vector component. What this % actually does is a non-linear transformation: % x_new = log(x_old - m + 1) % where m=min(x_old) and log is the natural logarithm. % Applying the transformation to a value which is lower % than m-1 will give problems, as the result is then complex. % If the minimum for values is known a priori, % it might be a good idea to initialize the normalization with % [dummy,sN] = som_norm_variable(minimum,'log','init'); % and normalize only after this: % x_new = som_norm_variable(x,sN,'do'); % % 'logistic' or softmax normalization. This normalization ensures % that all values in the future, too, are within the range % [0,1]. The transformation is more-or-less linear in the % middle range (around mean value), and has a smooth % nonlinearity at both ends which ensures that all values % are within the range. The data is first scaled as in % variance normalization: % x_scaled = (x_old - mean(x_old))/std(x_old) % and then transformed with the logistic function % x_new = 1/(1+exp(-x_scaled)) % % 'histD' Discrete histogram equalization. Non-linear. Orders the % values and replaces each value by its ordinal number. % Finally, scales the values such that they are between [0,1]. % Useful for both discrete and continuous variables, but as % the saved normalization information consists of all % unique values of the initialization data set, it may use % considerable amounts of memory. If the variable can get % more than a few values (say, 20), it might be better to % use 'histC' method below. Another important note is that % this method is not exactly revertible if it is applied % to values which are not part of the original value set. % % 'histC' Continuous histogram equalization. Actually, a partially % linear transformation which tries to do something like % histogram equalization. The value range is divided to % a number of bins such that the number of values in each % bin is (almost) the same. The values are transformed % linearly in each bin. For example, values in bin number 3 % are scaled between [3,4[. Finally, all values are scaled % between [0,1]. The number of bins is the square root % of the number of unique values in the initialization set, % rounded up. The resulting histogram equalization is not % as good as the one that 'histD' makes, but the benefit % is that it is exactly revertible - even outside the % original value range (although the results may be funny). % % 'eval' With this method, freeform normalization operations can be % specified. The parameter field contains strings to be % evaluated with 'eval' function, with variable name 'x' % representing the variable itself. The first string is % the normalization operation, and the second is a % denormalization operation. If the denormalization operation % is empty, it is ignored. % % INPUT ARGUMENTS % % x (vector) The scalar values to which the normalization % operation is applied. % % method The normalization specification. % (string) Identifier for a normalization method: 'var', % 'range', 'log', 'logistic', 'histD' or 'histC'. % Corresponding default normalization struct is created. % (struct) normalization struct % (struct array) of normalization structs, applied to % x one after the other % (cellstr) of length % (cellstr array) first string gives normalization operation, and % the second gives denormalization operation, with x % representing the variable, for example: % {'x+2','x-2}, or {'exp(-x)','-log(x)'} or {'round(x)'}. % Note that in the last case, no denorm operation is % defined. % % note: if the method is given as struct(s), it is % applied (done or undone, as specified by operation) % regardless of what the value of '.status' field % is in the struct(s). Only if the status is % 'uninit', the undoing operation is halted. % Anyhow, the '.status' fields in the returned % normalization struct(s) is set to approriate value. % % operation (string) The operation to perform: 'init' to initialize % the normalization struct, 'do' to perform the % normalization, 'undo' to undo the normalization, % if possible. If operation 'do' is given, but the % normalization struct has not yet been initialized, % it is initialized using the given data (x). % % OUTPUT ARGUMENTS % % x (vector) Appropriately processed values. % % sNorm (struct) Updated normalization struct/struct array. If any, % the '.status' and '.params' fields are updated. % % EXAMPLES % % To initialize and apply a normalization on a set of scalar values: % % [x_new,sN] = som_norm_variable(x_old,'var','do'); % % To just initialize, use: % % [dummy,sN] = som_norm_variable(x_old,'var','init'); % % To undo the normalization(s): % % x_orig = som_norm_variable(x_new,sN,'undo'); % % Typically, normalizations of data structs/sets are handled using % functions SOM_NORMALIZE and SOM_DENORMALIZE. However, when only the % values of a single variable are of interest, SOM_NORM_VARIABLE may % be useful. For example, assume one wants to apply the normalization % done on a component (i) of a data struct (sD) to a new set of values % (x) of that component. With SOM_NORM_VARIABLE this can be done with: % % x_new = som_norm_variable(x,sD.comp_norm{i},'do'); % % Now, as the normalizations in sD.comp_norm{i} have already been % initialized with the original data set (presumably sD.data), % the EXACTLY SAME normalization(s) can be applied to the new values. % The same thing can be done with SOM_NORMALIZE function, too: % % x_new = som_normalize(x,sD.comp_norm{i}); % % Or, if the new data set were in variable D - a matrix of same % dimension as the original data set: % % D_new = som_normalize(D,sD,i); % % SEE ALSO % % som_normalize Add/apply/redo normalizations for a data struct/set. % som_denormalize Undo normalizations of a data struct/set. % Copyright (c) 1998-2000 by the SOM toolbox programming team. % http://www.cis.hut.fi/projects/somtoolbox/ % Version 2.0beta juuso 151199 170400 150500 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% check arguments error(nargchk(3, 3, nargin)); % check no. of input arguments is correct % method sNorm = []; if ischar(method) if any(strcmp(method,{'var','range','log','logistic','histD','histC'})), sNorm = som_set('som_norm','method',method); else method = cellstr(method); end end if iscell(method), if length(method)==1 & isstruct(method{1}), sNorm = method{1}; else if length(method)==1 | isempty(method{2}), method{2} = 'x'; end sNorm = som_set('som_norm','method','eval','params',method); end else sNorm = method; end %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% action order = [1:length(sNorm)]; if length(order)>1 & strcmp(operation,'undo'), order = order(end:-1:1); end for i=order, % initialize if strcmp(operation,'init') | ... (strcmp(operation,'do') & strcmp(sNorm(i).status,'uninit')), % case method = 'hist' if strcmp(sNorm(i).method,'hist'), inds = find(~isnan(x) & ~isinf(x)); if length(unique(x(inds)))>20, sNorm(i).method = 'histC'; else sNorm{i}.method = 'histD'; end end switch(sNorm(i).method), case 'var', params = norm_variance_init(x); case 'range', params = norm_scale01_init(x); case 'log', params = norm_log_init(x); case 'logistic', params = norm_logistic_init(x); case 'histD', params = norm_histeqD_init(x); case 'histC', params = norm_histeqC_init(x); case 'eval', params = sNorm(i).params; otherwise, error(['Unrecognized method: ' sNorm(i).method]); end sNorm(i).params = params; sNorm(i).status = 'undone'; end % do / undo if strcmp(operation,'do'), switch(sNorm(i).method), case 'var', x = norm_scale_do(x,sNorm(i).params); case 'range', x = norm_scale_do(x,sNorm(i).params); case 'log', x = norm_log_do(x,sNorm(i).params); case 'logistic', x = norm_logistic_do(x,sNorm(i).params); case 'histD', x = norm_histeqD_do(x,sNorm(i).params); case 'histC', x = norm_histeqC_do(x,sNorm(i).params); case 'eval', x = norm_eval_do(x,sNorm(i).params); otherwise, error(['Unrecognized method: ' sNorm(i).method]); end sNorm(i).status = 'done'; elseif strcmp(operation,'undo'), if strcmp(sNorm(i).status,'uninit'), warning('Could not undo: uninitialized normalization struct.') break; end switch(sNorm(i).method), case 'var', x = norm_scale_undo(x,sNorm(i).params); case 'range', x = norm_scale_undo(x,sNorm(i).params); case 'log', x = norm_log_undo(x,sNorm(i).params); case 'logistic', x = norm_logistic_undo(x,sNorm(i).params); case 'histD', x = norm_histeqD_undo(x,sNorm(i).params); case 'histC', x = norm_histeqC_undo(x,sNorm(i).params); case 'eval', x = norm_eval_undo(x,sNorm(i).params); otherwise, error(['Unrecognized method: ' sNorm(i).method]); end sNorm(i).status = 'undone'; elseif ~strcmp(operation,'init'), error(['Unrecognized operation: ' operation]) end end return; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% subfunctions % linear scaling function p = norm_variance_init(x) inds = find(~isnan(x) & isfinite(x)); p = [mean(x(inds)), std(x(inds))]; if p(2) == 0, p(2) = 1; end %end of norm_variance_init function p = norm_scale01_init(x) inds = find(~isnan(x) & isfinite(x)); mi = min(x(inds)); ma = max(x(inds)); if mi == ma, p = [mi, 1]; else p = [mi, ma-mi]; end %end of norm_scale01_init function x = norm_scale_do(x,p) x = (x - p(1)) / p(2); % end of norm_scale_do function x = norm_scale_undo(x,p) x = x * p(2) + p(1); % end of norm_scale_undo % logarithm function p = norm_log_init(x) inds = find(~isnan(x) & isfinite(x)); p = min(x(inds)); % end of norm_log_init function x = norm_log_do(x,p) x = log(x - p +1); % if any(~isreal(x)), ok = 0; end % end of norm_log_do function x = norm_log_undo(x,p) x = exp(x) -1 + p; % end of norm_log_undo % logistic function p = norm_logistic_init(x) inds = find(~isnan(x) & isfinite(x)); p = [mean(x(inds)), std(x(inds))]; if p(2)==0, p(2) = 1; end % end of norm_logistic_init function x = norm_logistic_do(x,p) x = (x-p(1))/p(2); x = 1./(1+exp(-x)); % end of norm_logistic_do function x = norm_logistic_undo(x,p) x = log(x./(1-x)); x = x*p(2)+p(1); % end of norm_logistic_undo % histogram equalization for discrete values function p = norm_histeqD_init(x) inds = find(~isnan(x) & ~isinf(x)); p = unique(x(inds)); % end of norm_histeqD_init function x = norm_histeqD_do(x,p) bins = length(p); inds = find(~isnan(x) & ~isinf(x))'; for i = inds, [dummy ind] = min(abs(x(i) - p)); % data item closer to the left-hand bin wall is indexed after RH wall if x(i) > p(ind) & ind < bins, x(i) = ind + 1; else x(i) = ind; end end x = (x-1)/(bins-1); % normalization between [0,1] % end of norm_histeqD_do function x = norm_histeqD_undo(x,p) bins = length(p); x = round(x*(bins-1)+1); inds = find(~isnan(x) & ~isinf(x)); x(inds) = p(x(inds)); % end of norm_histeqD_undo % histogram equalization with partially linear functions function p = norm_histeqC_init(x) % investigate x inds = find(~isnan(x) & ~isinf(x)); samples = length(inds); xs = unique(x(inds)); mi = xs(1); ma = xs(end); % decide number of limits lims = ceil(sqrt(length(xs))); % 2->2,100->10,1000->32,10000->100 % decide limits if lims==1, p = [mi, mi+1]; lims = 2; elseif lims==2, p = [mi, ma]; else p = zeros(lims,1); p(1) = mi; p(end) = ma; binsize = zeros(lims-1,1); b = 1; avebinsize = samples/(lims-1); for i=1:(length(xs)-1), binsize(b) = binsize(b) + sum(x==xs(i)); if binsize(b) >= avebinsize, b = b + 1; p(b) = (xs(i)+xs(i+1))/2; end if b==(lims-1), binsize(b) = samples-sum(binsize); break; else avebinsize = (samples-sum(binsize))/(lims-1-b); end end end % end of norm_histeqC_init function x = norm_histeqC_do(x,p) xnew = x; lims = length(p); % handle values below minimum r = p(2)-p(1); inds = find(x<=p(1) & isfinite(x)); if any(inds), xnew(inds) = 0-(p(1)-x(inds))/r; end % handle values above maximum r = p(end)-p(end-1); inds = find(x>p(end) & isfinite(x)); if any(inds), xnew(inds) = lims-1+(x(inds)-p(end))/r; end % handle all other values for i=1:(lims-1), r0 = p(i); r1 = p(i+1); r = r1-r0; inds = find(x>r0 & x<=r1); if any(inds), xnew(inds) = i-1+(x(inds)-r0)/r; end end % scale so that minimum and maximum correspond to 0 and 1 x = xnew/(lims-1); % end of norm_histeqC_do function x = norm_histeqC_undo(x,p) xnew = x; lims = length(p); % scale so that 0 and 1 correspond to minimum and maximum x = x*(lims-1); % handle values below minimum r = p(2)-p(1); inds = find(x<=0 & isfinite(x)); if any(inds), xnew(inds) = x(inds)*r + p(1); end % handle values above maximum r = p(end)-p(end-1); inds = find(x>lims-1 & isfinite(x)); if any(inds), xnew(inds) = (x(inds)-(lims-1))*r+p(end); end % handle all other values for i=1:(lims-1), r0 = p(i); r1 = p(i+1); r = r1-r0; inds = find(x>i-1 & x<=i); if any(inds), xnew(inds) = (x(inds)-(i-1))*r + r0; end end x = xnew; % end of norm_histeqC_undo % eval function p = norm_eval_init(method) p = method; %end of norm_eval_init function x = norm_eval_do(x,p) x_tmp = eval(p{1}); if size(x_tmp,1)>=1 & size(x,1)>=1 & ... size(x_tmp,2)==1 & size(x,2)==1, x = x_tmp; end %end of norm_eval_do function x = norm_eval_undo(x,p) x_tmp = eval(p{2}); if size(x_tmp,1)>=1 & size(x,1)>=1 & ... size(x_tmp,2)==1 & size(x,2)==1, x = x_tmp; end %end of norm_eval_undo %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Powered by GitList 0.6.0