Package cssutils :: Module tokenize2 :: Class Tokenizer
[hide private]
[frames] | no frames]

Class Tokenizer

source code

object --+
         |
        Tokenizer

generates a list of Token tuples:
(Tokenname, value, startline, startcolumn)


Instance Methods [hide private]
 
_expand_macros(self, macros, productions)
returns macro expanded productions, order of productions is kept
source code
 
_compile_productions(self, expanded_productions)
compile productions into callable match objects, order is kept
source code
 
__init__(self, macros=None, productions=None)
inits tokenizer with given macros and productions which default to cssutils own macros and productions
source code
 
tokenize(self, text, fullsheet=False)
generator: tokenizes text and yiels tokens, each token is a tuple of:
source code

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Class Variables [hide private]
  _linesep = u'\n'
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, macros=None, productions=None)
(Constructor)

source code 
inits tokenizer with given macros and productions which default to cssutils own macros and productions
Overrides: object.__init__

tokenize(self, text, fullsheet=False)

source code 

generator: tokenizes text and yiels tokens, each token is a tuple of:

(tokenname, tokenvalue, line, col)

The tokenvalue will contain a normal string, meaning CSS unicode escapes have been resolved to normal characters. The serializer escapes needed characters back to unicode escapes depending of the stylesheet target encoding.

text
to be tokenized
fullsheet
if True appends EOF token as last one and completes incomplete COMMENT tokens